Nov 02 13:40:25.595865 ip-10-0-128-62 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 13:40:25.595879 ip-10-0-128-62 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 13:40:25.595886 ip-10-0-128-62 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 13:40:25.596108 ip-10-0-128-62 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 13:40:35.733837 ip-10-0-128-62 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 13:40:35.733854 ip-10-0-128-62 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 43ca23b7b60d4dd8b49f97c909c96d24 --
Nov 02 13:42:29.691845 ip-10-0-128-62 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:42:30.114013 ip-10-0-128-62 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:30.114013 ip-10-0-128-62 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:42:30.114013 ip-10-0-128-62 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:30.114013 ip-10-0-128-62 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:30.120267 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120182    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.120267 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120265    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.120267 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120269    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.120267 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120272    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.120267 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120275    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120278    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120281    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120283    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120286    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120288    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120291    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120293    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120298    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120302    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120305    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120307    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120311    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120315    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120318    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120321    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120324    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120326    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120329    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120331    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.120452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120334    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120343    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120346    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120349    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120351    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120354    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120356    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120359    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120361    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120364    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120366    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120368    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120371    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120374    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120377    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120379    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120382    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120385    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120387    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.120914 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120389    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120392    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120394    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120397    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120399    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120402    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120404    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120408    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120410    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120414    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120416    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120419    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120421    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120423    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120426    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120428    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120431    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120434    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120436    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.121338 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120439    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120441    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120444    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120446    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120449    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120451    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120454    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120456    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120459    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120462    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120465    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120467    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120470    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120472    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120474    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120476    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120479    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120482    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120484    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120486    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.121781 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120489    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120492    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120511    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120515    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120519    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120524    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120526    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120529    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120531    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120534    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120537    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120539    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120542    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120545    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120547    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120550    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120552    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120554    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120557    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120560    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.122232 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120562    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120565    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120567    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.120570    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121333    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121340    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121343    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121347    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121350    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121352    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121355    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121358    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121360    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121362    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121365    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121368    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121370    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121373    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121375    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121378    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.122766 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121380    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121382    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121385    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121389    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121392    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121394    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121397    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121400    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121402    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121404    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121407    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121409    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121412    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121414    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121416    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121419    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121421    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121424    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121426    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121428    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.123234 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121431    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121434    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121436    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121438    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121441    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121443    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121446    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121448    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121451    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121453    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121455    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121458    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121460    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121463    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121466    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121469    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121472    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121474    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121476    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121479    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.123741 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121482    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121486    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121489    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121492    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121510    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121513    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121516    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121518    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121521    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121524    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121526    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121529    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121532    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121536    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121538    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121541    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121544    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121547    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121549    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121552    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.124195 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121555    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121558    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121560    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121563    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121566    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121568    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121571    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121574    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121576    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121578    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121581    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121583    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121586    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121588    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121590    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121593    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121595    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121598    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121600    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.124664 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121603    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121605    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121608    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121610    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121613    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121615    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121618    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121620    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121624    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121626    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.121629    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122084    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122089    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122092    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122096    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122098    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122101    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122103    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122106    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.125095 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122109    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122111    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122114    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122116    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122119    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122121    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122123    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122126    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122128    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122131    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122133    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122136    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122138    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122140    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122143    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122146    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122148    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122150    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122152    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122155    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.125578 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122158    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122161    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122163    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122166    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122168    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122171    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122173    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122177    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122179    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122183    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122186    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122189    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122192    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122195    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122199    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122202    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122204    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122207    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122209    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.126034 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122212    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122216    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122219    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122221    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122224    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122227    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122230    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122232    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122235    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122237    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122240    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122242    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122245    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122248    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122250    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122254    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122258    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122261    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.126466 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122263    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122266    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122268    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122271    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122273    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122276    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122278    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122281    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122283    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122286    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122288    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122291    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122293    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122295    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122300    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122302    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122304    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122307    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122311    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122314    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.126995 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122316    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122319    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122321    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122323    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122326    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122328    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122331    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122333    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122335    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122338    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122340    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122342    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122345    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122347    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122349    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122352    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122355    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122357    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122360    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122362    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.122365    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.127461 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123328    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123342    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123350    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123354    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123358    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123362    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123366    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123370    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123373    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123377    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123380    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123383    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123386    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123389    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123392    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123394    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123397    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123400    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123404    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123406    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123409    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123412    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123416    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123419    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:42:30.127965 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123422    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123425    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123428    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123431    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123435    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123438    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123441    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123444    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123448    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123451    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123454    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123457    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123460    2575 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123463    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123467    2575 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123470    2575 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123473    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123476    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123479    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123483    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123485    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123488    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123491    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123510    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123515    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:42:30.128538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123519    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123522    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123524    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123527    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123530    2575 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123534    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123537    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123540    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123543    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123546    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123549    2575 flags.go:64] FLAG: --help="false"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123552    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123573    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123576    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123579    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123583    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123586    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123589    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123592    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123594    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123597    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123600    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123603    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123606    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:42:30.129122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123609    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123611    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123614    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123617    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123620    2575 flags.go:64] FLAG: --lock-file=""
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123623    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123625    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123628    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123642    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123644    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123647    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123650    2575 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123653    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123656    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123659    2575 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123662    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123666    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123669    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123673    2575 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123676    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123681    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123684    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123687    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123690    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123693    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:42:30.129890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123696    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123703    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123706    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123709    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123725    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123728    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123730    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123733    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123736    2575 flags.go:64] FLAG: --port="10250"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123738    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123741    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00624b1e8dfb8c6fb"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123744    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123747    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123750    2575 flags.go:64] FLAG: --register-node="true"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123753    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123757    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123759    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123762    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123764    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123768    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123771    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123773    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123776    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123779    2575 flags.go:64] FLAG: --runonce="false"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123782    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123785    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123788    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:42:30.131448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123792    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123795    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123800    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123803    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123806    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123809    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123812    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123814    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123817    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123820    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123823    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123826    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123830    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123833    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123836    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123840    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123843    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123845    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123848    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123851    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123854    2575 flags.go:64] FLAG: --v="2"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123858    2575 flags.go:64] FLAG: --version="false"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123862    2575 flags.go:64] FLAG: --vmodule=""
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123865    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.123869    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:42:30.132488 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.123992    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.123996    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.123999    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124002    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124004    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124007    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124009    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124013    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124016    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124019    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124022    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124026    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124030    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124033    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124036    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124040    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124044    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124047    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124050    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.133335 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124053    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124055    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124058    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124060    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124062    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124065    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124067    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124069    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124072    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124074    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124077    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124079    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124082    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124084    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124086    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124089    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124091    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124094    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124096    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124098    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.134150 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124103    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124106    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124110    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124112    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124115    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124118    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124120    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124123    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124125    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124128    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124131    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124133    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124136    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124138    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124141    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124143    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124146    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124148    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124151    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124153    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.135047 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124155    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124158    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124160    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124163    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124165    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124168    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124170    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124173    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124175    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124178    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124180    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124183    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124186    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124189    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124192    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124195    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124197    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124200    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124203    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.135661 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124206    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124208    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124211    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124213    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124215    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124218    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124221    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124223    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124225    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124228    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124230    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124233    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124235    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124237    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124240    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124242    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124244    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124247    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124249    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124251    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.136377 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124254    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124257    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124259    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124261    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124264    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124268    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124270    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124274    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.124299    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124331    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124336    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124339    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124342    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124345    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124347    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124350    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.137197 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124352    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124355    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124357    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124360    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124362    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124365    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124367    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124370    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124373    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124375    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124377    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124380    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124382    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124384    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124387    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124389    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124392    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124394    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124396    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124399    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.137615 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124401    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124405    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124408    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124410    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124414    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124416    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124419    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124421    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124424    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124426    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124429    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124432    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124434    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124436    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124439    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124441    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124444    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124446    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124449    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.138199 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124451    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124453    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124456    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124458    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124461    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124463    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124466    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124468    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124470    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124474    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124478    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124481    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124484    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124487    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124492    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124509    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124514    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124519    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124522    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.138691 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124525    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124527    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124530    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124533    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124536    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124538    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124541    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124543    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124545    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124548    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124550    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124553    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124555    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124558    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124561    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124563    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124565    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124568    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124570    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124573    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.139189 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124575    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124577    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124580    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124583    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124585    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124587    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124590    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124594    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124596    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124599    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124603    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124606    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124608    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124611    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124614    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124616    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124619    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124621    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124624    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.139733 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124627    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.124629    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.132290    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.132307    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.133003    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.133016    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133121    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133129    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133135    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133140    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133146    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133151    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133157    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133161    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133166    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133170    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133174    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133177    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133181    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.140196 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133186    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133190    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133194    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133198    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133205    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133210    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133214    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133218    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133222    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133227    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133239    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133243    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133247    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133251    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133255    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133259    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133264    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133269    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133273    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.140684 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133277    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133281    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133285    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133289    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133293    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133297    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133302    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133306    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133310    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133314    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133321    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133326    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133330    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133335    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133339    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133343    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133347    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133351    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133355    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.141435 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133359    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133363    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133367    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133371    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133376    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133387    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133392    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133396    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133400    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133404    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133408    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133413    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133417    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133421    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133425    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133429    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133434    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133438    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133442    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.142275 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133446    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133449    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133454    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133458    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133462    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133467    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133472    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133476    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133480    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133484    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133488    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133492    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133512    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133517    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133520    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133525    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133529    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133533    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133537    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133550    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.143090 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133555    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133559    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133563    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133567    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133572    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133583    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133587    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133591    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133595    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133599    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133603    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133607    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133611    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133615    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133619    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133623    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.143893 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.133665    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133949    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133958    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133963    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133968    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133972    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133976    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133980    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133984    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133988    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133992    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.133996    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134000    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134004    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134009    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134013    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134027    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134031    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134035    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134039    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134043    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.144470 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134047    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134051    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134055    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134059    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134063    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134067    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134070    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134074    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134078    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134082    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134106    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134113    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134119    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134124    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134128    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134132    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134137    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134141    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134145    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.145156 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134149    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134153    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134156    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134160    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134164    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134168    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134172    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134178    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134185    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134200    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134205    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134209    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134214    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134218    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134222    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134226    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134230    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134233    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134237    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.145648 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134242    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134245    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134249    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134253    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134257    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134261    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134265    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134269    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134273    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134277    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134281    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134285    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134289    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134293    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134297    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134301    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134305    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134309    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134314    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134318    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.146082 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134322    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134326    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134330    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134341    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134346    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134350    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134354    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134358    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134362    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134366    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134369    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134373    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134378    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134382    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134385    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134389    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134394    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134398    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134402    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.146618 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134406    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134411    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134417    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134421    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134425    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134430    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134435    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134439    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134443    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.134479    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134561    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134568    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134573    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134577    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134581    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134585    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:30.147087 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134589    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134607    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134611    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134615    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134619    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134624    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134628    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134649    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134653    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134658    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134662    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134665    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134670    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134675    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134685    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134689    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134694    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134698    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134702    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:30.147452 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134706    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134710    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134715    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134719    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134723    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134727    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134731    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134738    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134744    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134749    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134755    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134759    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134764    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134768    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134772    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134783    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134787    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134791    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134795    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:30.147913 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134799    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134803    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134808    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134812    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134816    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134820    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134824    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134828    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134832    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134836    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134841    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134845    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134849    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134853    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134858    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134862    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134866    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134870    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134874    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134879    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:30.148369 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134882    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134886    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134891    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134895    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134899    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134903    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134907    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134911    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134915    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134926    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134930    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134934    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134938    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134946    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134950    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134954    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134958    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134962    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134966    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:30.148864 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134971    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134977    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134983    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134988    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134992    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.134996    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135000    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135004    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135008    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135012    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135017    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135021    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135025    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135029    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135033    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135037    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135041    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135045    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135049    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135053    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:30.149293 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135056    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135060    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:30.135064    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.135153    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.137834    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.138833    2575 server.go:1014] "Starting client certificate rotation"
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.138937    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.139811    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.141448    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:42:30.149755 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.144899    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:42:30.154646 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.154613    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:42:30.157842 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.157821    2575 log.go:25] "Validated CRI v1 image API"
Nov 02 13:42:30.158692 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.158661    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:42:30.161628 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.161600    2575 fs.go:135] Filesystem UUIDs: map[33daa21b-7504-4a21-af30-ebd5d61aeca2:/dev/nvme0n1p4 5b332bc9-d2c7-4ad4-a14a-982a1a07b270:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 02 13:42:30.161704 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.161627    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 13:42:30.165913 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.165811    2575 manager.go:217] Machine: {Timestamp:2026-11-02 13:42:30.165349923 +0000 UTC m=+0.357814013 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198999 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0f6d90ea04eb67a993d917877980 SystemUUID:ec2d0f6d-90ea-04eb-67a9-93d917877980 BootID:43ca23b7-b60d-4dd8-b49f-97c909c96d24 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:84:d0:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:84:d0:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:5c:a9:bc:ba:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:42:30.165913 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.165906    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:42:30.166005 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.165973    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:42:30.169215 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.169191    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:42:30.169335 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.169216    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-62.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:42:30.169997 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.169984    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:42:30.170028 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.170001    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:42:30.170028 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.170013    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:42:30.170028 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.170026    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:42:30.170806 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.170796    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:42:30.173954 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.173937    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:42:30.174517 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.174491    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:42:30.174672 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.174662    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:42:30.174721 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.174692    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:42:30.174721 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.174713    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:42:30.174813 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.174732    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:42:30.174813 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.174745    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:42:30.174813 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.174756    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:42:30.175124 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.175111    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:42:30.177399 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.177386    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:42:30.182653 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.182620    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:42:30.183583 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183569    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:42:30.183656 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183609    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:42:30.183815 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183801    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:42:30.183872 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183822    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:42:30.183872 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183833    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:42:30.183872 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183842    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:42:30.183872 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183850    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:42:30.183872 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.183860    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:42:30.184338 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.184329    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:42:30.184385 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.184354    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:42:30.184385 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.184367    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:42:30.184385 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.184376    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:42:30.184528 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.184395    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:42:30.184528 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.184409    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:42:30.185172 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.185161    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:42:30.185221 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.185176    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:42:30.189577 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.189555    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-62.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:42:30.190038 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.190023    2575 server.go:1263] "Started kubelet"
Nov 02 13:42:30.190108 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.190056    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:42:30.190192 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.190112    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:42:30.190252 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.190219    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:42:30.190733 ip-10-0-128-62 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:42:30.192051 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.192036    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:42:30.194057 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.194038    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:42:30.194788 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.194769    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 13:42:30.197732 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.197716    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:42:30.198231 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.198215    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:42:30.198418 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.198385    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.198889 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.198872    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:42:30.198889 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.198876    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:42:30.199020 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.198900    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:42:30.199020 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.198994    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:42:30.199020 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199006    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:42:30.199196 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199166    2575 factory.go:55] Registering systemd factory
Nov 02 13:42:30.199196 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199180    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:42:30.199414 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199402    2575 factory.go:153] Registering CRI-O factory
Nov 02 13:42:30.199414 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199415    2575 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:42:30.199485 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199458    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:42:30.199485 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199477    2575 factory.go:103] Registering Raw factory
Nov 02 13:42:30.199574 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.199493    2575 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:42:30.200488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.200474    2575 manager.go:319] Starting recovery of all containers
Nov 02 13:42:30.202621 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.202588    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-62.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 13:42:30.202730 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.202700    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-62.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 13:42:30.202854 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.202716    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zjqkj"
Nov 02 13:42:30.203122 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.203095    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 13:42:30.203910 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.202837    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.us-east-2.compute.internal.18e43e3549a42d31  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 13:42:30.189985073 +0000 UTC m=+0.382449163,LastTimestamp:2026-11-02 13:42:30.189985073 +0000 UTC m=+0.382449163,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.us-east-2.compute.internal,}"
Nov 02 13:42:30.206209 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.206189    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 13:42:30.213603 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.213582    2575 manager.go:324] Recovery completed
Nov 02 13:42:30.214127 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.214106    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zjqkj"
Nov 02 13:42:30.217970 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.217955    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:30.219568 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.219551    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:30.219627 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.219575    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:30.219627 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.219585    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:42:30.220145 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.220130    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:42:30.220145 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.220141    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:42:30.220145 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.220144    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:30.220308 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.220155    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:42:30.220882 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.220865    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:30.220946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.220895    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:30.220946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.220906    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:42:30.222141 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.222131    2575 policy_none.go:50] "Start"
Nov 02 13:42:30.222193 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.222145    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:42:30.222193 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.222156    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:42:30.222475 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.222413    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.us-east-2.compute.internal.18e43e354b678c2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-62.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 13:42:30.219566125 +0000 UTC m=+0.412030215,LastTimestamp:2026-11-02 13:42:30.219566125 +0000 UTC m=+0.412030215,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.us-east-2.compute.internal,}"
Nov 02 13:42:30.224217 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.224207    2575 policy_none.go:44] "Start"
Nov 02 13:42:30.260035 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.260013    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:42:30.262394 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.262384    2575 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:42:30.262997 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.262985    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:42:30.262997 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.262997    2575 server.go:87] "Starting device plugin registration server"
Nov 02 13:42:30.263185 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.263176    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:42:30.263213 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.263189    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:42:30.263322 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.263302    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:42:30.263420 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.263402    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:42:30.263420 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.263418    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:42:30.263867 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.263835    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:42:30.263931 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.263869    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.329515 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.329481    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:42:30.331375 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.330905    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:42:30.331375 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.330924    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:42:30.331375 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.330942    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:42:30.331375 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.330970    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 13:42:30.334815 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.334796    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:30.363840 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.363827    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:30.364605 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.364570    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:30.364605 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.364596    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:30.364682 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.364608    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:42:30.364682 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.364630    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.372571 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.372410    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.372627 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.372580    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-62.us-east-2.compute.internal\": node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.385267 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.385246    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.431259 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.431232    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"]
Nov 02 13:42:30.431359 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.431287    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:30.432080 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.432066    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:30.432156 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.432097    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:30.432156 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.432112    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:42:30.434272 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.434258    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:30.434422 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.434408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.434467 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.434447    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:30.435034 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.435013    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:30.435034 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.435021    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:30.435117 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.435044    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:30.435117 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.435055    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:42:30.435117 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.435046    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:30.435201 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.435126    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:42:30.437099 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.437086    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.437158 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.437110    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:30.437784 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.437767    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:30.437857 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.437797    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:30.437857 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.437807    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:42:30.475885 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.475869    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found" node="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.478460 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.478444    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found" node="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.486323 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.486304    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.499616 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.499597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.499683 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.499621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.499683 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.499638    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4ea240f21eef62637ef08b4bbcf8f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"b4ea240f21eef62637ef08b4bbcf8f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.587389 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.587369    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.600641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.600622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.600699 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.600646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.600699 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.600661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4ea240f21eef62637ef08b4bbcf8f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"b4ea240f21eef62637ef08b4bbcf8f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.600763 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.600709    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4ea240f21eef62637ef08b4bbcf8f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"b4ea240f21eef62637ef08b4bbcf8f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.600763 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.600724    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.600763 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.600711    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.688076 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.688024    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.779554 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.779529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.782625 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:30.782614    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:30.788117 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.788102    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.888613 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.888585    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:30.989167 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:30.989110    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:31.089559 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:31.089539    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:31.139078 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.139060    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 13:42:31.139418 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.139179    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 13:42:31.190563 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:31.190543    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:42:31.219308 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.219273    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:37:31 +0000 UTC" deadline="2028-05-22 10:49:27.555865445 +0000 UTC"
Nov 02 13:42:31.219308 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.219303    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13605h6m56.336565439s"
Nov 02 13:42:31.236883 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.236863    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:31.264590 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.264544    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:31.288985 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.288933    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:31.299300 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.299278    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:31.310577 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.310557    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:42:31.311644 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.311620    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:42:31.322772 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.322756    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:42:31.405879 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:31.405857    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ea240f21eef62637ef08b4bbcf8f6a.slice/crio-eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc WatchSource:0}: Error finding container eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc: Status 404 returned error can't find the container with id eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc
Nov 02 13:42:31.406215 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:31.406191    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848b57801b929e97e838cbabe1dcb19f.slice/crio-87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30 WatchSource:0}: Error finding container 87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30: Status 404 returned error can't find the container with id 87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30
Nov 02 13:42:31.410392 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:31.410371    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:42:33.664293 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.664259    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:33.734825 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.734801    2575 apiserver.go:52] "Watching apiserver"
Nov 02 13:42:33.744254 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.744220    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:42:33.744494 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.744477    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-jrphq","openshift-cluster-node-tuning-operator/tuned-wsn55","openshift-ovn-kubernetes/ovnkube-node-2qbkw","openshift-dns/node-resolver-sfqfd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-gbsrr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht","openshift-multus/multus-grrwm","openshift-multus/network-metrics-daemon-l7vld","openshift-network-operator/iptables-alerter-g2xz6","kube-system/global-pull-secret-syncer-q6z7m","kube-system/konnectivity-agent-jt476","kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal","openshift-image-registry/node-ca-cxttx"]
Nov 02 13:42:33.747740 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.747023    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.749282 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.749266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.751552 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.751532    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:42:33.751666 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.751532    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:42:33.751666 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.751536    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:42:33.751666 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.751630    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8rzz2\""
Nov 02 13:42:33.751816 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.751668    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:42:33.751816 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.751696    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:42:33.751906 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.751843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.753783 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.753762    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:42:33.753783 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.753762    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:42:33.753934 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.753913    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fbcvm\""
Nov 02 13:42:33.754067 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.754052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:33.754145 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.754125    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:33.756229 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756206    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:42:33.756316 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756287    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:42:33.756371 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756348    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:42:33.756429 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756391    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:42:33.756485 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756465    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:42:33.756485 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756472    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:42:33.756602 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756538    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjbpx\""
Nov 02 13:42:33.756602 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.756544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.758324 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.758296    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 13:42:33.758547 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.758530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:33.758645 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.758618    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:33.761674 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.761650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.765127 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.764912    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-systemd\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766587    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-var-lib-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-bin\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766642    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnj9d\" (UniqueName: \"kubernetes.io/projected/33b41081-1ac8-414a-9911-450a0b7b7d7f-kube-api-access-hnj9d\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-cni-binary-copy\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27t78\" (UniqueName: \"kubernetes.io/projected/ca506817-08d9-420e-b25d-45dc0de360c4-kube-api-access-27t78\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-etc-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766807    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-system-cni-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-run\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.766885 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-sys\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-kubelet\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-node-log\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-log-socket\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.766998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-script-lib\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-socket-dir-parent\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-hostroot\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-var-lib-kubelet\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-tuned\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-ovn\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-env-overrides\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767191    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-etc-kubernetes\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzxj\" (UniqueName: \"kubernetes.io/projected/9c6359e8-e292-457f-b680-f6b44bee7e79-kube-api-access-rgzxj\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767303    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-cnibin\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.767416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-kubelet\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-conf-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-daemon-config\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-lib-modules\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-host\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-systemd-units\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-k8s-cni-cncf-io\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-slash\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysconfig\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hl9\" (UniqueName: \"kubernetes.io/projected/68685f74-14d9-444a-bbf1-9292f78d6a8e-kube-api-access-24hl9\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767866    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-cnibin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.767931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.767898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-bin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768019    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-netns\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-systemd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-multus\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-netns\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovn-node-metrics-cert\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-modprobe-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-conf\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-config\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768302    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768323    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768344    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-multus-certs\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-kubernetes\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjzw\" (UniqueName: \"kubernetes.io/projected/8b376c48-407e-4913-8b07-a0eb3fcc67a0-kube-api-access-5wjzw\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-netd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.768513 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-system-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.769200 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768521    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-tmp\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.769200 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.768815    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.769200 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.768878    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:33.775733 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.772948    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:33.775895 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.775845    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.792198 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.792160    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:42:33.793886 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.792220    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-crsj9"
Nov 02 13:42:33.804144 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.804124    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-crsj9"
Nov 02 13:42:33.851776 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.851752    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:42:33.851776 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.851762    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:42:33.851957 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.851775    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:42:33.851957 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.851815    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:42:33.851957 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.851831    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:42:33.851957 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.851841    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:42:33.852148 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.852005    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:42:33.852148 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.852033    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:42:33.853231 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853215    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:42:33.853414 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853399    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:42:33.853488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853396    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mlv7f\""
Nov 02 13:42:33.853488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853437    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:42:33.853488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853481    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:42:33.853664 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853522    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n56j5\""
Nov 02 13:42:33.853664 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853552    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:42:33.853664 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853574    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tb7ws\""
Nov 02 13:42:33.853664 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853634    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cfdrs\""
Nov 02 13:42:33.853867 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853669    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cmphd\""
Nov 02 13:42:33.853867 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853668    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s2t6n\""
Nov 02 13:42:33.853867 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.853682    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:42:33.859498 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.859480    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:42:33.869379 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:33.869468 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.869468 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysconfig\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.869468 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hl9\" (UniqueName: \"kubernetes.io/projected/68685f74-14d9-444a-bbf1-9292f78d6a8e-kube-api-access-24hl9\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:33.869468 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-cnibin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.869641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-bin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.869641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869511    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvvp\" (UniqueName: \"kubernetes.io/projected/b672addc-8b6e-4983-bc93-17ff035c88bf-kube-api-access-kkvvp\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.869641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869534    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-dbus\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.869641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.869641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-netns\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.869641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-systemd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-multus\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsd7\" (UniqueName: \"kubernetes.io/projected/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-kube-api-access-snsd7\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-netns\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-serviceca\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovn-node-metrics-cert\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-modprobe-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-conf\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-config\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.869946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-multus-certs\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.869984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-kubernetes\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjzw\" (UniqueName: \"kubernetes.io/projected/8b376c48-407e-4913-8b07-a0eb3fcc67a0-kube-api-access-5wjzw\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-netd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-system-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70301f22-b14c-4362-a082-15309eafa41b-agent-certs\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870108    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nltd\" (UniqueName: \"kubernetes.io/projected/7b652b34-c239-4b27-ada2-ec7b39d17539-kube-api-access-9nltd\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-tmp\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73189526-2e64-4de5-a2a9-06597d293b6b-tmp-dir\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-device-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-systemd\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-var-lib-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-bin\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnj9d\" (UniqueName: \"kubernetes.io/projected/33b41081-1ac8-414a-9911-450a0b7b7d7f-kube-api-access-hnj9d\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-cni-binary-copy\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.870598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70301f22-b14c-4362-a082-15309eafa41b-konnectivity-ca\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-sys-fs\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27t78\" (UniqueName: \"kubernetes.io/projected/ca506817-08d9-420e-b25d-45dc0de360c4-kube-api-access-27t78\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73189526-2e64-4de5-a2a9-06597d293b6b-hosts-file\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-etc-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-host\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-system-cni-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxqj\" (UniqueName: \"kubernetes.io/projected/73189526-2e64-4de5-a2a9-06597d293b6b-kube-api-access-hnxqj\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b672addc-8b6e-4983-bc93-17ff035c88bf-iptables-alerter-script\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-run\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-sys\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-kubelet\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870758    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-node-log\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-log-socket\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-script-lib\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.871250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-socket-dir-parent\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-hostroot\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-var-lib-kubelet\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-tuned\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-ovn\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-env-overrides\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.870997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-etc-kubernetes\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzxj\" (UniqueName: \"kubernetes.io/projected/9c6359e8-e292-457f-b680-f6b44bee7e79-kube-api-access-rgzxj\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b672addc-8b6e-4983-bc93-17ff035c88bf-host-slash\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-cnibin\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-kubelet\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-conf-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-daemon-config\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-kubelet-config\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-lib-modules\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.871989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-host\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-systemd-units\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-k8s-cni-cncf-io\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-socket-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-slash\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.871416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-registration-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872148    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872252    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysconfig\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872471    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-cnibin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872504    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-bin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872674    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-netns\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872700    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-systemd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.872738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872733    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.873399 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872761    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-multus\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.873399 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.872794    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-netns\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.873399 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873069    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:42:33.873399 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873144    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-modprobe-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.873399 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873253    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-conf\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.873627 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873443    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-sys\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.873627 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873490    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-kubelet\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.873627 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873530    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-node-log\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.873627 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873586    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-log-socket\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.873804 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873739    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-netd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.873804 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873792    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-system-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.873929 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873815    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-config\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.873929 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873877    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-systemd\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.874024 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873932    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-run\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.874024 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.873976    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-etc-kubernetes\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.874024 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874017    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-socket-dir-parent\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.874148 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874056    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-hostroot\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.874148 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874073    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-script-lib\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.874148 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874101    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-var-lib-kubelet\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.874313 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874295    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-k8s-cni-cncf-io\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.874375 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874348    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-host\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.874433 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874384    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-systemd-units\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.874433 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.874528 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874466    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-slash\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.874528 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874515    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.874666 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874550    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-var-lib-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.874666 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874595    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-ovn\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.874932 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.874913    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-env-overrides\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.875032 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.875004    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-lib-modules\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.875107 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.875042    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-cnibin\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.875331 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.875255    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.875395 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.875356    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-kubelet\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.875395 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.875384    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-conf-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.875868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.875846    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-daemon-config\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.876284 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.876262    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.876411 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.876373    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.876467 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.876448    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:33.876516 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.876503    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs podName:68685f74-14d9-444a-bbf1-9292f78d6a8e nodeName:}" failed. No retries permitted until 2026-11-02 13:42:34.376485571 +0000 UTC m=+3.009888371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs") pod "network-metrics-daemon-l7vld" (UID: "68685f74-14d9-444a-bbf1-9292f78d6a8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:33.876891 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.876872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-bin\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.877490 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877467    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-cni-binary-copy\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.877743 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877722    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-tuned\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.877837 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877780    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-tmp\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.877837 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877818    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-etc-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.877946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877865    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-kubernetes\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.877946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877866    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-multus-certs\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.877946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877877    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.877946 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.877899    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-system-cni-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.878216 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.878191    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovn-node-metrics-cert\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.886948 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.886929    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:33.887038 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.886952    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:33.887038 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.886965    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjbql for pod openshift-network-diagnostics/network-check-target-gbsrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:33.887038 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.887012    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql podName:91cccc0e-d0fc-4b34-badd-f4efbf8fd684 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:34.3869972 +0000 UTC m=+3.020399997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vjbql" (UniqueName: "kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql") pod "network-check-target-gbsrr" (UID: "91cccc0e-d0fc-4b34-badd-f4efbf8fd684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:33.893207 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.893179    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjzw\" (UniqueName: \"kubernetes.io/projected/8b376c48-407e-4913-8b07-a0eb3fcc67a0-kube-api-access-5wjzw\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:33.893544 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.893526    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzxj\" (UniqueName: \"kubernetes.io/projected/9c6359e8-e292-457f-b680-f6b44bee7e79-kube-api-access-rgzxj\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:42:33.893769 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.893744    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hl9\" (UniqueName: \"kubernetes.io/projected/68685f74-14d9-444a-bbf1-9292f78d6a8e-kube-api-access-24hl9\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:33.893846 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.893773    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnj9d\" (UniqueName: \"kubernetes.io/projected/33b41081-1ac8-414a-9911-450a0b7b7d7f-kube-api-access-hnj9d\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:33.893929 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.893912    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27t78\" (UniqueName: \"kubernetes.io/projected/ca506817-08d9-420e-b25d-45dc0de360c4-kube-api-access-27t78\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:33.899622 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.899583    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"b4ea240f21eef62637ef08b4bbcf8f6a","Type":"ContainerStarted","Data":"eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc"}
Nov 02 13:42:33.900880 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.900859    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"848b57801b929e97e838cbabe1dcb19f","Type":"ContainerStarted","Data":"87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30"}
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-host\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxqj\" (UniqueName: \"kubernetes.io/projected/73189526-2e64-4de5-a2a9-06597d293b6b-kube-api-access-hnxqj\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972634    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b672addc-8b6e-4983-bc93-17ff035c88bf-iptables-alerter-script\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b672addc-8b6e-4983-bc93-17ff035c88bf-host-slash\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-kubelet-config\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972706    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-host\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-socket-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-registration-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvvp\" (UniqueName: \"kubernetes.io/projected/b672addc-8b6e-4983-bc93-17ff035c88bf-kube-api-access-kkvvp\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-dbus\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972848    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-socket-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsd7\" (UniqueName: \"kubernetes.io/projected/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-kube-api-access-snsd7\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-serviceca\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972938    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70301f22-b14c-4362-a082-15309eafa41b-agent-certs\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nltd\" (UniqueName: \"kubernetes.io/projected/7b652b34-c239-4b27-ada2-ec7b39d17539-kube-api-access-9nltd\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.973403 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.972984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73189526-2e64-4de5-a2a9-06597d293b6b-tmp-dir\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-device-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70301f22-b14c-4362-a082-15309eafa41b-konnectivity-ca\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-sys-fs\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973093    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-kubelet-config\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73189526-2e64-4de5-a2a9-06597d293b6b-hosts-file\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973201    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b672addc-8b6e-4983-bc93-17ff035c88bf-iptables-alerter-script\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973261    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973272    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-device-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973057    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b672addc-8b6e-4983-bc93-17ff035c88bf-host-slash\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973340    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-sys-fs\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973389    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73189526-2e64-4de5-a2a9-06597d293b6b-hosts-file\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973436    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-dbus\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973479    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-registration-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.973516    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.973872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73189526-2e64-4de5-a2a9-06597d293b6b-tmp-dir\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.974868 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.974091    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.975821 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:33.974151    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret podName:71c1e4ef-71b9-4a62-b841-a21453d83e59 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:34.47354959 +0000 UTC m=+3.106952376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret") pod "global-pull-secret-syncer-q6z7m" (UID: "71c1e4ef-71b9-4a62-b841-a21453d83e59") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:33.975821 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.974893    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-serviceca\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:33.975821 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.975071    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70301f22-b14c-4362-a082-15309eafa41b-konnectivity-ca\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:33.976336 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.976301    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70301f22-b14c-4362-a082-15309eafa41b-agent-certs\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:33.985792 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.985772    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvvp\" (UniqueName: \"kubernetes.io/projected/b672addc-8b6e-4983-bc93-17ff035c88bf-kube-api-access-kkvvp\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:33.985951 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.985934    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nltd\" (UniqueName: \"kubernetes.io/projected/7b652b34-c239-4b27-ada2-ec7b39d17539-kube-api-access-9nltd\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:33.986026 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.986001    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxqj\" (UniqueName: \"kubernetes.io/projected/73189526-2e64-4de5-a2a9-06597d293b6b-kube-api-access-hnxqj\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:33.993458 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:33.993441    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsd7\" (UniqueName: \"kubernetes.io/projected/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-kube-api-access-snsd7\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:34.075337 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.075272    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:42:34.085494 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.085471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:42:34.085848 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.085819    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca506817_08d9_420e_b25d_45dc0de360c4.slice/crio-e70b3b6d1ee0b55591a26e3b5da81fe703c2f5d1d542a14606b327bb09be8f19 WatchSource:0}: Error finding container e70b3b6d1ee0b55591a26e3b5da81fe703c2f5d1d542a14606b327bb09be8f19: Status 404 returned error can't find the container with id e70b3b6d1ee0b55591a26e3b5da81fe703c2f5d1d542a14606b327bb09be8f19
Nov 02 13:42:34.094394 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.094371    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b376c48_407e_4913_8b07_a0eb3fcc67a0.slice/crio-cec1b8bc1caf35c875cf6a1e3e6c48e478a5c1220cac1154ec7df0a6285de89c WatchSource:0}: Error finding container cec1b8bc1caf35c875cf6a1e3e6c48e478a5c1220cac1154ec7df0a6285de89c: Status 404 returned error can't find the container with id cec1b8bc1caf35c875cf6a1e3e6c48e478a5c1220cac1154ec7df0a6285de89c
Nov 02 13:42:34.094986 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.094969    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grrwm"
Nov 02 13:42:34.096112 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.096092    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:34.103729 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.103712    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:42:34.105491 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.105472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:42:34.105998 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.105980    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6359e8_e292_457f_b680_f6b44bee7e79.slice/crio-0390309ce3f675d966a964da3efa3a78c1da668837cc046ea37bdd874b969d9a WatchSource:0}: Error finding container 0390309ce3f675d966a964da3efa3a78c1da668837cc046ea37bdd874b969d9a: Status 404 returned error can't find the container with id 0390309ce3f675d966a964da3efa3a78c1da668837cc046ea37bdd874b969d9a
Nov 02 13:42:34.106686 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.106631    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b41081_1ac8_414a_9911_450a0b7b7d7f.slice/crio-2a3dea0116d7928578aa74c6ef9a427180ba71a8d4716145f9cf741641d1137c WatchSource:0}: Error finding container 2a3dea0116d7928578aa74c6ef9a427180ba71a8d4716145f9cf741641d1137c: Status 404 returned error can't find the container with id 2a3dea0116d7928578aa74c6ef9a427180ba71a8d4716145f9cf741641d1137c
Nov 02 13:42:34.116906 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.116831    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b652b34_c239_4b27_ada2_ec7b39d17539.slice/crio-9f13ce4dc3a40098afa1318defbbc3bd3d77288494ed65e6a116a5465a4105c1 WatchSource:0}: Error finding container 9f13ce4dc3a40098afa1318defbbc3bd3d77288494ed65e6a116a5465a4105c1: Status 404 returned error can't find the container with id 9f13ce4dc3a40098afa1318defbbc3bd3d77288494ed65e6a116a5465a4105c1
Nov 02 13:42:34.118871 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.118849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:42:34.119157 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.119136    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73189526_2e64_4de5_a2a9_06597d293b6b.slice/crio-c7c0fc1931916b9511c1ce75cfbf7b26dacd2444935d229f233392d2b9ec5e86 WatchSource:0}: Error finding container c7c0fc1931916b9511c1ce75cfbf7b26dacd2444935d229f233392d2b9ec5e86: Status 404 returned error can't find the container with id c7c0fc1931916b9511c1ce75cfbf7b26dacd2444935d229f233392d2b9ec5e86
Nov 02 13:42:34.127241 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.126916    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:34.128678 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.128652    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb672addc_8b6e_4983_bc93_17ff035c88bf.slice/crio-1dae972d8b96164201c2d90321d585deca175e8ae96015e1f04603dcdfaf0eb1 WatchSource:0}: Error finding container 1dae972d8b96164201c2d90321d585deca175e8ae96015e1f04603dcdfaf0eb1: Status 404 returned error can't find the container with id 1dae972d8b96164201c2d90321d585deca175e8ae96015e1f04603dcdfaf0eb1
Nov 02 13:42:34.132123 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.132089    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:42:34.136815 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.136793    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70301f22_b14c_4362_a082_15309eafa41b.slice/crio-62a68096776c43d66201f8f2b671e5eb981807dba25263666710e31a1bc73b22 WatchSource:0}: Error finding container 62a68096776c43d66201f8f2b671e5eb981807dba25263666710e31a1bc73b22: Status 404 returned error can't find the container with id 62a68096776c43d66201f8f2b671e5eb981807dba25263666710e31a1bc73b22
Nov 02 13:42:34.142477 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:42:34.142450    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8ef4a9_bc67_43d4_a30a_2bcd08bbbeea.slice/crio-15f87d486fc614b1378b929466f0e8fc74494592f7d06bdeccd942824cb3be1c WatchSource:0}: Error finding container 15f87d486fc614b1378b929466f0e8fc74494592f7d06bdeccd942824cb3be1c: Status 404 returned error can't find the container with id 15f87d486fc614b1378b929466f0e8fc74494592f7d06bdeccd942824cb3be1c
Nov 02 13:42:34.476969 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.476907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:34.476969 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.476951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:34.477166 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.476977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:34.477166 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477068    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:34.477166 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477069    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:34.477166 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477074    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:34.477166 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477137    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:34.477166 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477151    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjbql for pod openshift-network-diagnostics/network-check-target-gbsrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:34.477367 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477117    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret podName:71c1e4ef-71b9-4a62-b841-a21453d83e59 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:35.477101385 +0000 UTC m=+4.110504166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret") pod "global-pull-secret-syncer-q6z7m" (UID: "71c1e4ef-71b9-4a62-b841-a21453d83e59") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:34.477367 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477203    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs podName:68685f74-14d9-444a-bbf1-9292f78d6a8e nodeName:}" failed. No retries permitted until 2026-11-02 13:42:35.477186102 +0000 UTC m=+4.110588898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs") pod "network-metrics-daemon-l7vld" (UID: "68685f74-14d9-444a-bbf1-9292f78d6a8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:34.477367 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:34.477219    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql podName:91cccc0e-d0fc-4b34-badd-f4efbf8fd684 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:35.477210305 +0000 UTC m=+4.110613093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjbql" (UniqueName: "kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql") pod "network-check-target-gbsrr" (UID: "91cccc0e-d0fc-4b34-badd-f4efbf8fd684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:34.808219 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.808180    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:37:33 +0000 UTC" deadline="2028-04-03 07:38:02.734797208 +0000 UTC"
Nov 02 13:42:34.808219 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.808216    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12425h55m27.926584753s"
Nov 02 13:42:34.919729 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.919693    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"b4ea240f21eef62637ef08b4bbcf8f6a","Type":"ContainerStarted","Data":"b5a9dae04a5a9557ed3fb40da55b1c43ac2f7b90112dfe03332b04b20819917a"}
Nov 02 13:42:34.945594 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.943348    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jt476" event={"ID":"70301f22-b14c-4362-a082-15309eafa41b","Type":"ContainerStarted","Data":"62a68096776c43d66201f8f2b671e5eb981807dba25263666710e31a1bc73b22"}
Nov 02 13:42:34.948983 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.948920    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2xz6" event={"ID":"b672addc-8b6e-4983-bc93-17ff035c88bf","Type":"ContainerStarted","Data":"1dae972d8b96164201c2d90321d585deca175e8ae96015e1f04603dcdfaf0eb1"}
Nov 02 13:42:34.950360 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.950293    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht" event={"ID":"7b652b34-c239-4b27-ada2-ec7b39d17539","Type":"ContainerStarted","Data":"9f13ce4dc3a40098afa1318defbbc3bd3d77288494ed65e6a116a5465a4105c1"}
Nov 02 13:42:34.952718 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.952660    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wsn55" event={"ID":"8b376c48-407e-4913-8b07-a0eb3fcc67a0","Type":"ContainerStarted","Data":"cec1b8bc1caf35c875cf6a1e3e6c48e478a5c1220cac1154ec7df0a6285de89c"}
Nov 02 13:42:34.956130 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.956105    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxttx" event={"ID":"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea","Type":"ContainerStarted","Data":"15f87d486fc614b1378b929466f0e8fc74494592f7d06bdeccd942824cb3be1c"}
Nov 02 13:42:34.964790 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.964767    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sfqfd" event={"ID":"73189526-2e64-4de5-a2a9-06597d293b6b","Type":"ContainerStarted","Data":"c7c0fc1931916b9511c1ce75cfbf7b26dacd2444935d229f233392d2b9ec5e86"}
Nov 02 13:42:34.971586 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.971509    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"2a3dea0116d7928578aa74c6ef9a427180ba71a8d4716145f9cf741641d1137c"}
Nov 02 13:42:34.975232 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.975177    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grrwm" event={"ID":"9c6359e8-e292-457f-b680-f6b44bee7e79","Type":"ContainerStarted","Data":"0390309ce3f675d966a964da3efa3a78c1da668837cc046ea37bdd874b969d9a"}
Nov 02 13:42:34.979750 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:34.979733    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerStarted","Data":"e70b3b6d1ee0b55591a26e3b5da81fe703c2f5d1d542a14606b327bb09be8f19"}
Nov 02 13:42:35.490448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.490207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:35.490682 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.490486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:35.490682 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.490521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:35.490814 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.490688    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:35.490814 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.490747    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret podName:71c1e4ef-71b9-4a62-b841-a21453d83e59 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:37.490729621 +0000 UTC m=+6.124132410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret") pod "global-pull-secret-syncer-q6z7m" (UID: "71c1e4ef-71b9-4a62-b841-a21453d83e59") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:35.491155 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.491136    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:35.491212 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.491165    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:35.491212 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.491177    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjbql for pod openshift-network-diagnostics/network-check-target-gbsrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:35.491298 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.491222    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql podName:91cccc0e-d0fc-4b34-badd-f4efbf8fd684 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:37.491207514 +0000 UTC m=+6.124610300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjbql" (UniqueName: "kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql") pod "network-check-target-gbsrr" (UID: "91cccc0e-d0fc-4b34-badd-f4efbf8fd684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:35.491298 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.491278    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:35.491436 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.491307    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs podName:68685f74-14d9-444a-bbf1-9292f78d6a8e nodeName:}" failed. No retries permitted until 2026-11-02 13:42:37.491297208 +0000 UTC m=+6.124699991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs") pod "network-metrics-daemon-l7vld" (UID: "68685f74-14d9-444a-bbf1-9292f78d6a8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:35.808653 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.808501    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:37:33 +0000 UTC" deadline="2028-05-17 21:45:19.581994636 +0000 UTC"
Nov 02 13:42:35.808653 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.808571    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13496h2m43.77344219s"
Nov 02 13:42:35.896316 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.896291    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:35.896472 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.896432    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:35.896914 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.896657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:35.896914 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.896748    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:35.896914 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.896804    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:35.896914 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:35.896871    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:35.998764 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.998731    2575 generic.go:356] "Generic (PLEG): container finished" podID="848b57801b929e97e838cbabe1dcb19f" containerID="6eb9f9faab78374ef6ea61a2e7a0b4981fcca1d0b49a5577f2407c204384f3f2" exitCode=0
Nov 02 13:42:35.998930 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:35.998803    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"848b57801b929e97e838cbabe1dcb19f","Type":"ContainerDied","Data":"6eb9f9faab78374ef6ea61a2e7a0b4981fcca1d0b49a5577f2407c204384f3f2"}
Nov 02 13:42:36.017524 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:36.017471    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal" podStartSLOduration=4.017454114 podStartE2EDuration="4.017454114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:34.943933002 +0000 UTC m=+3.577335804" watchObservedRunningTime="2026-11-02 13:42:36.017454114 +0000 UTC m=+4.650856920"
Nov 02 13:42:37.007597 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.007509    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"848b57801b929e97e838cbabe1dcb19f","Type":"ContainerStarted","Data":"b27e5a2024ae1714427b6838f1fe701e9f2af8dc117b4bcc3fe1c060a83222dd"}
Nov 02 13:42:37.021367 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.021338    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerStarted","Data":"e15a2d3efac5d82736b3d8f28f32e40d52c8382dfb298059d559dcc51bb3320f"}
Nov 02 13:42:37.066410 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.066356    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal" podStartSLOduration=5.066342922 podStartE2EDuration="5.066342922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:37.027580716 +0000 UTC m=+5.660983523" watchObservedRunningTime="2026-11-02 13:42:37.066342922 +0000 UTC m=+5.699745724"
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.506935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.506991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.507010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507120    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507167    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret podName:71c1e4ef-71b9-4a62-b841-a21453d83e59 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:41.507153983 +0000 UTC m=+10.140556778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret") pod "global-pull-secret-syncer-q6z7m" (UID: "71c1e4ef-71b9-4a62-b841-a21453d83e59") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507512    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507528    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507540    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjbql for pod openshift-network-diagnostics/network-check-target-gbsrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:37.507626 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507597    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql podName:91cccc0e-d0fc-4b34-badd-f4efbf8fd684 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:41.50758327 +0000 UTC m=+10.140986066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjbql" (UniqueName: "kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql") pod "network-check-target-gbsrr" (UID: "91cccc0e-d0fc-4b34-badd-f4efbf8fd684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:37.508189 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507667    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:37.508189 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.507707    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs podName:68685f74-14d9-444a-bbf1-9292f78d6a8e nodeName:}" failed. No retries permitted until 2026-11-02 13:42:41.507695972 +0000 UTC m=+10.141098754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs") pod "network-metrics-daemon-l7vld" (UID: "68685f74-14d9-444a-bbf1-9292f78d6a8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:37.892851 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.892773    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:37.893002 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.892903    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:37.893287 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.893265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:37.893387 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.893362    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:37.893450 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:37.893408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:37.893488 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:37.893450    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:38.030601 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:38.030553    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="e15a2d3efac5d82736b3d8f28f32e40d52c8382dfb298059d559dcc51bb3320f" exitCode=0
Nov 02 13:42:38.031836 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:38.031622    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerDied","Data":"e15a2d3efac5d82736b3d8f28f32e40d52c8382dfb298059d559dcc51bb3320f"}
Nov 02 13:42:39.037134 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:39.037060    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jt476" event={"ID":"70301f22-b14c-4362-a082-15309eafa41b","Type":"ContainerStarted","Data":"a70b0545b103f832a71d599c4fcec948b9dd43dafba3bf66f3b258b156edc4f8"}
Nov 02 13:42:39.055259 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:39.055208    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jt476" podStartSLOduration=3.235214523 podStartE2EDuration="7.055191597s" totalImagesPullingTime="3.819977074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:39.055026847 +0000 UTC m=+7.688429651" watchObservedRunningTime="2026-11-02 13:42:39.055191597 +0000 UTC m=+7.688594403"
Nov 02 13:42:39.891441 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:39.890650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:39.891441 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:39.890793    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:39.891441 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:39.891258    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:39.891441 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:39.891395    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:39.891807 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:39.891656    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:39.891807 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:39.891760    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:40.044849 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:40.044769    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht" event={"ID":"7b652b34-c239-4b27-ada2-ec7b39d17539","Type":"ContainerStarted","Data":"e9841157f789a530e1046dd41ea4bdd5ee200fa0f739beb0e9f447fce549c43e"}
Nov 02 13:42:40.051171 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:40.051146    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxttx" event={"ID":"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea","Type":"ContainerStarted","Data":"add351240cf8d60c88cbd111ff7ae722ecbb648667ba4f4f2d79a7da5b50ccde"}
Nov 02 13:42:41.542405 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.542378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:41.542778 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.542429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:41.542778 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.542458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:41.542778 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.542587    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:41.542778 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.542645    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret podName:71c1e4ef-71b9-4a62-b841-a21453d83e59 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:49.542628126 +0000 UTC m=+18.176030921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret") pod "global-pull-secret-syncer-q6z7m" (UID: "71c1e4ef-71b9-4a62-b841-a21453d83e59") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:41.543248 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.543035    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:41.543248 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.543058    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:41.543248 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.543071    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjbql for pod openshift-network-diagnostics/network-check-target-gbsrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:41.543248 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.543119    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql podName:91cccc0e-d0fc-4b34-badd-f4efbf8fd684 nodeName:}" failed. No retries permitted until 2026-11-02 13:42:49.543103821 +0000 UTC m=+18.176506615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjbql" (UniqueName: "kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql") pod "network-check-target-gbsrr" (UID: "91cccc0e-d0fc-4b34-badd-f4efbf8fd684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:41.543248 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.543185    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:41.543248 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.543218    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs podName:68685f74-14d9-444a-bbf1-9292f78d6a8e nodeName:}" failed. No retries permitted until 2026-11-02 13:42:49.543206887 +0000 UTC m=+18.176609686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs") pod "network-metrics-daemon-l7vld" (UID: "68685f74-14d9-444a-bbf1-9292f78d6a8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:41.569061 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.569030    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:42:41.830639 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.830047    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:42:41.569050287Z","UUID":"a3024adc-23f7-411b-b0b5-14cdfa4b2240","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:42:41.831884 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.831859    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:42:41.831999 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.831911    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:42:41.892492 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.892443    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:41.892693 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.892606    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:41.892937 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.892918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:41.893038 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.893006    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:41.893100 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:41.893067    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:41.893166 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:41.893146    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:42.055861 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:42.055170    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sfqfd" event={"ID":"73189526-2e64-4de5-a2a9-06597d293b6b","Type":"ContainerStarted","Data":"798592ca75ee50dff82cf3d401d26561ec3c431ba9da02d8c5c06c6c075872cc"}
Nov 02 13:42:42.058050 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:42.058026    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grrwm" event={"ID":"9c6359e8-e292-457f-b680-f6b44bee7e79","Type":"ContainerStarted","Data":"e2e901a875ee80fb5c95a10ecfe032e717e889063aa9bd2e4c0f35dbe7cf6561"}
Nov 02 13:42:42.068068 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:42.068049    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht" event={"ID":"7b652b34-c239-4b27-ada2-ec7b39d17539","Type":"ContainerStarted","Data":"e3c04e424f31da44663d5f29eb0bfa2f6ed8340d6215198ab7d67b15e5074b66"}
Nov 02 13:42:42.076990 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:42.076946    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cxttx" podStartSLOduration=5.290062356 podStartE2EDuration="10.076930638s" totalImagesPullingTime="4.786868282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:40.072206738 +0000 UTC m=+8.705609545" watchObservedRunningTime="2026-11-02 13:42:42.076930638 +0000 UTC m=+10.710333443"
Nov 02 13:42:42.099760 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:42.099215    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sfqfd" podStartSLOduration=2.4802162389999998 podStartE2EDuration="10.099203589s" totalImagesPullingTime="7.61898735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:42.077412147 +0000 UTC m=+10.710814951" watchObservedRunningTime="2026-11-02 13:42:42.099203589 +0000 UTC m=+10.732606434"
Nov 02 13:42:42.099760 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:42.099469    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-grrwm" podStartSLOduration=3.50629573 podStartE2EDuration="11.099463209s" totalImagesPullingTime="7.593167479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:42.098477295 +0000 UTC m=+10.731880099" watchObservedRunningTime="2026-11-02 13:42:42.099463209 +0000 UTC m=+10.732866018"
Nov 02 13:42:43.072976 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.072867    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht" event={"ID":"7b652b34-c239-4b27-ada2-ec7b39d17539","Type":"ContainerStarted","Data":"1207d8493a660f347affdcee9d1d417bd01f047c7d36a6a88534c9e928e08600"}
Nov 02 13:42:43.075106 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.075078    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="96d37d63616744e46922f2d8e36fcdcadbff1f37248f025957717d805eff4422" exitCode=0
Nov 02 13:42:43.075218 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.075165    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerDied","Data":"96d37d63616744e46922f2d8e36fcdcadbff1f37248f025957717d805eff4422"}
Nov 02 13:42:43.122141 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.122082    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht" podStartSLOduration=2.900198791 podStartE2EDuration="11.122063316s" totalImagesPullingTime="8.221864525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:43.096246689 +0000 UTC m=+11.729649492" watchObservedRunningTime="2026-11-02 13:42:43.122063316 +0000 UTC m=+11.755466122"
Nov 02 13:42:43.700598 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.700327    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:43.701194 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.701171    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:43.890455 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.890432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:43.890455 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.890452    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:43.890638 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:43.890542    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:43.890638 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:43.890572    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:43.890638 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:43.890629    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:43.890820 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:43.890705    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:44.080182 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:44.080102    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="2ceaaba45b723a9bf6dc833075d09c310ae2b1edc77ccc6a345e377129bca1b0" exitCode=0
Nov 02 13:42:44.080940 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:44.080487    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerDied","Data":"2ceaaba45b723a9bf6dc833075d09c310ae2b1edc77ccc6a345e377129bca1b0"}
Nov 02 13:42:44.085349 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:44.085323    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g2xz6" event={"ID":"b672addc-8b6e-4983-bc93-17ff035c88bf","Type":"ContainerStarted","Data":"aa67353ac6eec61de70007273ffc9ef0e91d8d9c2bb715dcf2128c4b2b73dc1c"}
Nov 02 13:42:44.125265 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:44.125227    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g2xz6" podStartSLOduration=4.18907643 podStartE2EDuration="12.125216139s" totalImagesPullingTime="7.936139709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:44.125090535 +0000 UTC m=+12.758493338" watchObservedRunningTime="2026-11-02 13:42:44.125216139 +0000 UTC m=+12.758618939"
Nov 02 13:42:45.088931 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:45.088898    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wsn55" event={"ID":"8b376c48-407e-4913-8b07-a0eb3fcc67a0","Type":"ContainerStarted","Data":"64c7f0ee6596603bf8e3433a77a4ad8181ff0f7564d563128fe548986efd8d9f"}
Nov 02 13:42:45.110486 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:45.110442    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wsn55" podStartSLOduration=3.941156245 podStartE2EDuration="14.110429794s" totalImagesPullingTime="10.169273549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:45.110310802 +0000 UTC m=+13.743713605" watchObservedRunningTime="2026-11-02 13:42:45.110429794 +0000 UTC m=+13.743832597"
Nov 02 13:42:45.890531 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:45.890498    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:45.890531 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:45.890516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:45.890724 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:45.890544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:45.890724 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:45.890644    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:45.890724 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:45.890706    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:45.890831 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:45.890771    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:46.093744 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:46.093708    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="1f45c016b284939bd18d3d1a8b5a0e606845858ef0650da6e096ec0675ea0285" exitCode=0
Nov 02 13:42:46.094457 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:46.093796    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerDied","Data":"1f45c016b284939bd18d3d1a8b5a0e606845858ef0650da6e096ec0675ea0285"}
Nov 02 13:42:47.891214 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:47.891182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:47.891214 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:47.891205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:47.891657 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:47.891374    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:47.891657 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:47.891396    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:47.891657 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:47.891471    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:47.891657 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:47.891547    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:49.102327 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.101935    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"3a27d0f79c521462f99a55e9a52f66dc727ecee9a51faff6420980ffe5da7bf8"}
Nov 02 13:42:49.102818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.102335    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"8c722b4f68eea11657b82360c9db920385d4f07d7ed7769d1f614c1c53cc9f64"}
Nov 02 13:42:49.102818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.102387    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"b199284abad35328ef4adec1e163f9a94b8ad8bdca9df7dabefabbb04f1701d7"}
Nov 02 13:42:49.102818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.102428    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"7e6cb01e645964af4b23a9aeba7cff7fa966fa16e1c3948fbf359effedd8be0e"}
Nov 02 13:42:49.102818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.102472    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"69dab15d8a308bd173c809f9f51f614f830a5c0989a86a78d73f31e34321e16d"}
Nov 02 13:42:49.102818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.102507    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"11c6c789c0069783344a54cf12dc2c909f6ea7e0c9dd51e38d0abb127955301b"}
Nov 02 13:42:49.104747 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.104726    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="0cdfae6f5a9361612d6d05128b0aee33b5d46088d7f38478705e2e5000d92604" exitCode=0
Nov 02 13:42:49.104844 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.104754    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerDied","Data":"0cdfae6f5a9361612d6d05128b0aee33b5d46088d7f38478705e2e5000d92604"}
Nov 02 13:42:49.606252 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.606224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:49.606419 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.606272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:49.606419 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606364    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:49.606419 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606374    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:49.606419 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606390    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:49.606419 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606399    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjbql for pod openshift-network-diagnostics/network-check-target-gbsrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:49.606419 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606419    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs podName:68685f74-14d9-444a-bbf1-9292f78d6a8e nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.606401514 +0000 UTC m=+34.239804329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs") pod "network-metrics-daemon-l7vld" (UID: "68685f74-14d9-444a-bbf1-9292f78d6a8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:49.606731 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.606413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:49.606731 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606437    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql podName:91cccc0e-d0fc-4b34-badd-f4efbf8fd684 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.606427255 +0000 UTC m=+34.239830036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjbql" (UniqueName: "kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql") pod "network-check-target-gbsrr" (UID: "91cccc0e-d0fc-4b34-badd-f4efbf8fd684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:49.606731 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606484    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:49.606731 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.606526    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret podName:71c1e4ef-71b9-4a62-b841-a21453d83e59 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.60651657 +0000 UTC m=+34.239919368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret") pod "global-pull-secret-syncer-q6z7m" (UID: "71c1e4ef-71b9-4a62-b841-a21453d83e59") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:49.891253 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.891179    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:49.891253 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.891201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:49.891414 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:49.891267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:49.891414 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.891358    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:49.891502 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.891483    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:49.891599 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:49.891582    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:50.109651 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:50.109615    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="55ecae147b737da07d455827c81533633c262b87d784294648d8c3c5e468e285" exitCode=0
Nov 02 13:42:50.109982 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:50.109664    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerDied","Data":"55ecae147b737da07d455827c81533633c262b87d784294648d8c3c5e468e285"}
Nov 02 13:42:51.115087 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.115006    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"7118e274151c87425d7404f93b80b9660b967210c9bd8e39568ada09caa21e61"}
Nov 02 13:42:51.118263 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.118240    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrphq" event={"ID":"ca506817-08d9-420e-b25d-45dc0de360c4","Type":"ContainerStarted","Data":"226b709970f1d03b39724bbba879f55b244cfaef378aad77da8042ccc49b67e3"}
Nov 02 13:42:51.148315 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.148275    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jrphq" podStartSLOduration=9.20658009 podStartE2EDuration="20.148263532s" totalImagesPullingTime="10.941683442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:51.148070925 +0000 UTC m=+19.781473729" watchObservedRunningTime="2026-11-02 13:42:51.148263532 +0000 UTC m=+19.781666335"
Nov 02 13:42:51.313350 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.313329    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:51.313433 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.313426    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:42:51.313997 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.313962    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:42:51.891773 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.891744    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:51.891909 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:51.891826    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:51.891974 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.891914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:51.892006 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:51.891993    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:51.892044 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:51.892034    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:51.892096 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:51.892084    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:53.125832 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:53.125803    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" event={"ID":"33b41081-1ac8-414a-9911-450a0b7b7d7f","Type":"ContainerStarted","Data":"17962413f91fecaa9c4a3751b3ffbf4746d750fc14bb6d010d185c89d6a10074"}
Nov 02 13:42:53.126463 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:53.126021    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:53.142343 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:53.142296    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:53.163984 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:53.163942    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" podStartSLOduration=7.87303824 podStartE2EDuration="22.163930741s" totalImagesPullingTime="14.290892501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:53.163616164 +0000 UTC m=+21.797018967" watchObservedRunningTime="2026-11-02 13:42:53.163930741 +0000 UTC m=+21.797333545"
Nov 02 13:42:53.891133 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:53.891102    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:53.891276 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:53.891102    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:53.891276 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:53.891192    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:53.891276 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:53.891209    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:53.891388 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:53.891293    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:53.891388 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:53.891380    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:54.128267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.128240    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:42:54.134256 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.134219    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:54.150894 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.150823    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:54.654364 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.654189    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbsrr"]
Nov 02 13:42:54.654545 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.654447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:54.654616 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:54.654551    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:54.658003 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.657977    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q6z7m"]
Nov 02 13:42:54.658111 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.658031    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7vld"]
Nov 02 13:42:54.658111 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.658109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:54.658230 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:54.658208    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:54.658299 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:54.658283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:54.658382 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:54.658366    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:55.130945 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:55.130922    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:42:56.133468 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:56.133436    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:42:56.890279 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:56.890255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:56.890422 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:56.890256    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:56.890422 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:56.890349    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:56.890543 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:56.890256    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:56.890543 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:56.890432    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:56.890543 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:56.890532    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:42:57.258280 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:57.258207    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:42:57.258628 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:57.258436    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:42:57.275596 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:57.275544    2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" podUID="33b41081-1ac8-414a-9911-450a0b7b7d7f" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 13:42:57.284430 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:57.284398    2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw" podUID="33b41081-1ac8-414a-9911-450a0b7b7d7f" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 13:42:58.890595 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:58.890545    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:42:58.890595 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:58.890583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:42:58.891039 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:42:58.890669    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:42:58.891039 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:58.890687    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:42:58.891039 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:58.890742    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:42:58.891039 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:42:58.890804    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:00.472945 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:00.472921    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sfqfd_73189526-2e64-4de5-a2a9-06597d293b6b/dns-node-resolver/0.log"
Nov 02 13:43:00.890578 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:00.890532    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:00.890736 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:00.890535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:00.890736 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:00.890654    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:00.890831 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:00.890759    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:00.890831 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:00.890544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:00.890949 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:00.890907    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:01.441440 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:01.441417    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxttx_5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea/node-ca/0.log"
Nov 02 13:43:02.890734 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:02.890697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:02.891144 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:02.890709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:02.891144 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:02.890820    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:02.891144 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:02.890718    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:02.891144 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:02.890880    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:02.891144 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:02.890940    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:04.890470 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:04.890444    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:04.890856 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:04.890524    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:04.890856 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:04.890534    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:04.890856 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:04.890630    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:04.890856 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:04.890666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:04.890856 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:04.890716    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:05.616021 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:05.615990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:05.616021 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:05.616025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:05.616073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616141    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616163    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616176    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616187    2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjbql for pod openshift-network-diagnostics/network-check-target-gbsrr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616203    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs podName:68685f74-14d9-444a-bbf1-9292f78d6a8e nodeName:}" failed. No retries permitted until 2026-11-02 13:43:37.616186328 +0000 UTC m=+66.249589135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs") pod "network-metrics-daemon-l7vld" (UID: "68685f74-14d9-444a-bbf1-9292f78d6a8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616203    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616221    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql podName:91cccc0e-d0fc-4b34-badd-f4efbf8fd684 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:37.616209653 +0000 UTC m=+66.249612443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjbql" (UniqueName: "kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql") pod "network-check-target-gbsrr" (UID: "91cccc0e-d0fc-4b34-badd-f4efbf8fd684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:05.616287 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:05.616250    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret podName:71c1e4ef-71b9-4a62-b841-a21453d83e59 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:37.616236207 +0000 UTC m=+66.249638989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret") pod "global-pull-secret-syncer-q6z7m" (UID: "71c1e4ef-71b9-4a62-b841-a21453d83e59") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:06.891250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:06.891218    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:06.891250 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:06.891243    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:06.891768 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:06.891218    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:06.891768 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:06.891347    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:06.891768 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:06.891396    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:06.891768 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:06.891449    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:08.890904 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:08.890867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:08.890904 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:08.890892    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:08.891346 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:08.890979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:08.891346 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:08.890995    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:08.891346 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:08.891053    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:08.891346 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:08.891107    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:10.890961 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:10.890930    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:10.891338 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:10.891030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:10.891338 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:10.891038    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:10.891338 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:10.891059    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:10.891338 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:10.891136    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:10.891338 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:10.891205    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:12.890148 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:12.890120    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:12.890544 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:12.890203    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:12.890544 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:12.890276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:12.890544 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:12.890364    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:12.890544 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:12.890435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:12.890544 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:12.890502    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:14.891146 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:14.891101    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:14.891146 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:14.891128    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:14.891146 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:14.891143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:14.891669 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:14.891217    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6z7m" podUID="71c1e4ef-71b9-4a62-b841-a21453d83e59"
Nov 02 13:43:14.891669 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:14.891346    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7vld" podUID="68685f74-14d9-444a-bbf1-9292f78d6a8e"
Nov 02 13:43:14.891669 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:14.891385    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbsrr" podUID="91cccc0e-d0fc-4b34-badd-f4efbf8fd684"
Nov 02 13:43:16.664268 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.664242    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeReady"
Nov 02 13:43:16.664784 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.664394    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 13:43:16.726951 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.726918    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8gf9j"]
Nov 02 13:43:16.728628 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.728613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.737141 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.737122    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-29msq"]
Nov 02 13:43:16.738835 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.738814    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:43:16.738926 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.738843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.738926 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.738871    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:43:16.739115 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.739089    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:43:16.739115 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.739093    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:43:16.739213 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.739177    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-clq6z\""
Nov 02 13:43:16.746252 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.744383    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8gf9j"]
Nov 02 13:43:16.752610 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.752588    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-29msq"]
Nov 02 13:43:16.755249 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.755223    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vpzfd"]
Nov 02 13:43:16.756958 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.756944    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:16.773973 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.771876    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpzfd"]
Nov 02 13:43:16.841047 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.841003    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:43:16.841227 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.841139    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:43:16.841476 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.841461    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:43:16.841608 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.841593    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sw4zl\""
Nov 02 13:43:16.841678 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.841595    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cw7nr\""
Nov 02 13:43:16.841678 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.841627    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:43:16.841817 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.841804    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:43:16.891257 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.891221    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:16.891410 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.891268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:16.891410 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.891329    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:16.897079 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrfg\" (UniqueName: \"kubernetes.io/projected/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-api-access-6nrfg\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.897201 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-tmp-dir\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.897201 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2b48\" (UniqueName: \"kubernetes.io/projected/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-kube-api-access-x2b48\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.897201 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.897201 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q6p\" (UniqueName: \"kubernetes.io/projected/cf3f6767-bfe3-4c84-b337-e17679488a51-kube-api-access-97q6p\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:16.897339 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21c18bda-36de-4e16-9ea6-952ace591a1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.897339 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-metrics-tls\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.897406 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-config-volume\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.897406 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf3f6767-bfe3-4c84-b337-e17679488a51-cert\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:16.897406 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21c18bda-36de-4e16-9ea6-952ace591a1f-crio-socket\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.897503 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.897423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21c18bda-36de-4e16-9ea6-952ace591a1f-data-volume\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.942111 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.941972    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:43:16.942361 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.942132    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:43:16.942361 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.942352    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:43:16.943375 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.943357    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:43:16.943375 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.943369    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qngrq\""
Nov 02 13:43:16.943474 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.943390    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzjbd\""
Nov 02 13:43:16.943474 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.943395    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m5gp5\""
Nov 02 13:43:16.998648 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.998648 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q6p\" (UniqueName: \"kubernetes.io/projected/cf3f6767-bfe3-4c84-b337-e17679488a51-kube-api-access-97q6p\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:16.998890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21c18bda-36de-4e16-9ea6-952ace591a1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.998890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-metrics-tls\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.998890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-config-volume\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.998890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998827    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf3f6767-bfe3-4c84-b337-e17679488a51-cert\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:16.998890 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21c18bda-36de-4e16-9ea6-952ace591a1f-crio-socket\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.999121 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.998934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21c18bda-36de-4e16-9ea6-952ace591a1f-data-volume\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.999121 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrfg\" (UniqueName: \"kubernetes.io/projected/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-api-access-6nrfg\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.999121 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-tmp-dir\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.999121 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b48\" (UniqueName: \"kubernetes.io/projected/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-kube-api-access-x2b48\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.999319 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999167    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21c18bda-36de-4e16-9ea6-952ace591a1f-crio-socket\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.999373 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999312    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-config-volume\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:16.999419 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999387    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.999467 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999437    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21c18bda-36de-4e16-9ea6-952ace591a1f-data-volume\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:16.999641 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:16.999622    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-tmp-dir\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:17.002765 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.002743    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-metrics-tls\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:17.002882 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.002743    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21c18bda-36de-4e16-9ea6-952ace591a1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:17.002882 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.002835    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf3f6767-bfe3-4c84-b337-e17679488a51-cert\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:17.012060 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.012039    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q6p\" (UniqueName: \"kubernetes.io/projected/cf3f6767-bfe3-4c84-b337-e17679488a51-kube-api-access-97q6p\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:17.012350 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.012330    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2b48\" (UniqueName: \"kubernetes.io/projected/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-kube-api-access-x2b48\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:17.013715 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.013699    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrfg\" (UniqueName: \"kubernetes.io/projected/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-api-access-6nrfg\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:17.040122 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.040093    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:17.049918 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.049839    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:17.067659 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.067636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:17.338168 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.338134    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-29msq"]
Nov 02 13:43:17.339449 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.339206    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8gf9j"]
Nov 02 13:43:17.342650 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:17.342626    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582ae8d7_d1b6_4066_a863_a14a5f2eb45f.slice/crio-0dc1303df8c6a2a326a300c97835a1ede77ada40caf150cd1fe5b8c68cb4deef WatchSource:0}: Error finding container 0dc1303df8c6a2a326a300c97835a1ede77ada40caf150cd1fe5b8c68cb4deef: Status 404 returned error can't find the container with id 0dc1303df8c6a2a326a300c97835a1ede77ada40caf150cd1fe5b8c68cb4deef
Nov 02 13:43:17.342888 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:17.342864    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c18bda_36de_4e16_9ea6_952ace591a1f.slice/crio-89fb3ac01094f442a7e0a52f0b7853253147c5772fb5159baf1c84c8c20fe55c WatchSource:0}: Error finding container 89fb3ac01094f442a7e0a52f0b7853253147c5772fb5159baf1c84c8c20fe55c: Status 404 returned error can't find the container with id 89fb3ac01094f442a7e0a52f0b7853253147c5772fb5159baf1c84c8c20fe55c
Nov 02 13:43:17.363861 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:17.363741    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpzfd"]
Nov 02 13:43:17.375666 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:17.375638    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3f6767_bfe3_4c84_b337_e17679488a51.slice/crio-253ef5f2b34461ff99fab354c090190b292a59d5c554408e35aa4ba5bc66f990 WatchSource:0}: Error finding container 253ef5f2b34461ff99fab354c090190b292a59d5c554408e35aa4ba5bc66f990: Status 404 returned error can't find the container with id 253ef5f2b34461ff99fab354c090190b292a59d5c554408e35aa4ba5bc66f990
Nov 02 13:43:18.192204 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:18.192177    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpzfd" event={"ID":"cf3f6767-bfe3-4c84-b337-e17679488a51","Type":"ContainerStarted","Data":"253ef5f2b34461ff99fab354c090190b292a59d5c554408e35aa4ba5bc66f990"}
Nov 02 13:43:18.194441 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:18.194413    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gf9j" event={"ID":"21c18bda-36de-4e16-9ea6-952ace591a1f","Type":"ContainerStarted","Data":"68db376ce5d21b1bc3c649657fdbeb002b4352d6340c87a361c425ce2e595a3e"}
Nov 02 13:43:18.194547 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:18.194465    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gf9j" event={"ID":"21c18bda-36de-4e16-9ea6-952ace591a1f","Type":"ContainerStarted","Data":"b027a5d671bef3c1408bc5f43c6dee5eb8e171602248644a71aedecdc72123a9"}
Nov 02 13:43:18.194547 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:18.194485    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gf9j" event={"ID":"21c18bda-36de-4e16-9ea6-952ace591a1f","Type":"ContainerStarted","Data":"89fb3ac01094f442a7e0a52f0b7853253147c5772fb5159baf1c84c8c20fe55c"}
Nov 02 13:43:18.196053 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:18.196028    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29msq" event={"ID":"582ae8d7-d1b6-4066-a863-a14a5f2eb45f","Type":"ContainerStarted","Data":"0dc1303df8c6a2a326a300c97835a1ede77ada40caf150cd1fe5b8c68cb4deef"}
Nov 02 13:43:20.098614 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.098584    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"]
Nov 02 13:43:20.100381 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.100367    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.108538 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.108458    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 13:43:20.108738 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.108721    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 13:43:20.108809 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.108753    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9j4gh\""
Nov 02 13:43:20.110037 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.110017    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 13:43:20.110037 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.110025    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 13:43:20.110255 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.110072    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 13:43:20.114918 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.114899    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"]
Nov 02 13:43:20.204821 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.204793    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29msq" event={"ID":"582ae8d7-d1b6-4066-a863-a14a5f2eb45f","Type":"ContainerStarted","Data":"6706c7440dace6f364da16a025cc6c7ba39ca5bd9cf51f448b2c1d187601756b"}
Nov 02 13:43:20.205013 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.204851    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29msq" event={"ID":"582ae8d7-d1b6-4066-a863-a14a5f2eb45f","Type":"ContainerStarted","Data":"59255a9cdb7780f1048ec90e7ace2cd053a9aff8eab78c4181744f2cb1a986a6"}
Nov 02 13:43:20.205013 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.204884    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:20.206465 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.206438    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpzfd" event={"ID":"cf3f6767-bfe3-4c84-b337-e17679488a51","Type":"ContainerStarted","Data":"1f8ce410fae4c47dadd796e8e702a4556c752e234a8ebb300cabfbbe199869c3"}
Nov 02 13:43:20.208481 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.208455    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8gf9j" event={"ID":"21c18bda-36de-4e16-9ea6-952ace591a1f","Type":"ContainerStarted","Data":"02b1dc38db20d6186f426519d6dc605f385d9586af866200df38eb4f958f3737"}
Nov 02 13:43:20.221171 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.221145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9fw\" (UniqueName: \"kubernetes.io/projected/aa65ead1-60e6-42c4-bc86-ef6df717b751-kube-api-access-9k9fw\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.221267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.221207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.221267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.221228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.221337 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.221308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa65ead1-60e6-42c4-bc86-ef6df717b751-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.254955 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.254907    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-29msq" podStartSLOduration=2.417174298 podStartE2EDuration="4.254893432s" totalImagesPullingTime="1.837719134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:20.234713845 +0000 UTC m=+48.868116650" watchObservedRunningTime="2026-11-02 13:43:20.254893432 +0000 UTC m=+48.888296230"
Nov 02 13:43:20.255084 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.254990    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vpzfd" podStartSLOduration=1.589726998 podStartE2EDuration="4.25498494s" totalImagesPullingTime="2.665257942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:20.254202538 +0000 UTC m=+48.887605342" watchObservedRunningTime="2026-11-02 13:43:20.25498494 +0000 UTC m=+48.888387744"
Nov 02 13:43:20.281540 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.281498    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8gf9j" podStartSLOduration=2.112317829 podStartE2EDuration="4.281486273s" totalImagesPullingTime="2.169168444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:20.281398153 +0000 UTC m=+48.914800956" watchObservedRunningTime="2026-11-02 13:43:20.281486273 +0000 UTC m=+48.914889089"
Nov 02 13:43:20.322531 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.322508    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.322816 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.322790    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa65ead1-60e6-42c4-bc86-ef6df717b751-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.322951 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.322900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9fw\" (UniqueName: \"kubernetes.io/projected/aa65ead1-60e6-42c4-bc86-ef6df717b751-kube-api-access-9k9fw\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.323366 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.323028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.323527 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.323502    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa65ead1-60e6-42c4-bc86-ef6df717b751-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.324992 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.324972    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.325303 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.325285    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.340651 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.340632    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9fw\" (UniqueName: \"kubernetes.io/projected/aa65ead1-60e6-42c4-bc86-ef6df717b751-kube-api-access-9k9fw\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.411484 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.411447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:20.683064 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:20.682996    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"]
Nov 02 13:43:20.685756 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:20.685734    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa65ead1_60e6_42c4_bc86_ef6df717b751.slice/crio-f02f0af1d313803870af6eeeeb0599183b0f3d8c805015a1cd2b353a76e7f625 WatchSource:0}: Error finding container f02f0af1d313803870af6eeeeb0599183b0f3d8c805015a1cd2b353a76e7f625: Status 404 returned error can't find the container with id f02f0af1d313803870af6eeeeb0599183b0f3d8c805015a1cd2b353a76e7f625
Nov 02 13:43:21.212079 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:21.212038    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr" event={"ID":"aa65ead1-60e6-42c4-bc86-ef6df717b751","Type":"ContainerStarted","Data":"f02f0af1d313803870af6eeeeb0599183b0f3d8c805015a1cd2b353a76e7f625"}
Nov 02 13:43:22.216675 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:22.216589    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr" event={"ID":"aa65ead1-60e6-42c4-bc86-ef6df717b751","Type":"ContainerStarted","Data":"e5cc664c17ccfeab77b1336369db306fc21b3399cab41ff28a83babc747bc057"}
Nov 02 13:43:22.217137 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:22.216686    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr" event={"ID":"aa65ead1-60e6-42c4-bc86-ef6df717b751","Type":"ContainerStarted","Data":"b281baed19cf8fe1140edf003ecf627c956369321065b365628b23f03e5411cc"}
Nov 02 13:43:22.240099 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:22.240047    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr" podStartSLOduration=1.025560998 podStartE2EDuration="2.240034384s" totalImagesPullingTime="1.214473386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:22.239035391 +0000 UTC m=+50.872438196" watchObservedRunningTime="2026-11-02 13:43:22.240034384 +0000 UTC m=+50.873437187"
Nov 02 13:43:24.511853 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.511822    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tczrv"]
Nov 02 13:43:24.515004 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.514989    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.575470 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.575441    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 13:43:24.576234 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.576219    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 13:43:24.576460 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.576449    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gnlzz\""
Nov 02 13:43:24.591736 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.591717    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 13:43:24.652994 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.652972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjdj\" (UniqueName: \"kubernetes.io/projected/ae0be5eb-c95f-462e-ab68-3e4dffaae950-kube-api-access-tmjdj\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653117 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-textfile\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653117 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-wtmp\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653187 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-tls\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653187 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-metrics-client-ca\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653243 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-sys\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653243 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-root\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653243 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.653330 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.653250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-accelerators-collector-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754529 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754504    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjdj\" (UniqueName: \"kubernetes.io/projected/ae0be5eb-c95f-462e-ab68-3e4dffaae950-kube-api-access-tmjdj\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754672 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-textfile\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754726 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-wtmp\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754759 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-tls\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754759 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-metrics-client-ca\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-sys\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-root\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754900 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754900 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-accelerators-collector-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754900 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754848    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-textfile\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754900 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754857    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-wtmp\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.754900 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754878    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-sys\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.755113 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.754923    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-root\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.755353 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.755333    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-accelerators-collector-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.755429 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.755387    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-metrics-client-ca\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.757208 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.757178    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-tls\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.757294 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.757241    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.780169 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.780120    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dd7f9986f-b8g5g"]
Nov 02 13:43:24.782809 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.782789    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjdj\" (UniqueName: \"kubernetes.io/projected/ae0be5eb-c95f-462e-ab68-3e4dffaae950-kube-api-access-tmjdj\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.783195 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.783182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.790332 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.790317    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 13:43:24.803886 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.803870    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 13:43:24.804124 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.804107    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 13:43:24.804124 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.804118    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 13:43:24.804256 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.804139    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 13:43:24.804256 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.804156    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 13:43:24.810125 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.810105    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 13:43:24.811052 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.811033    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd7f9986f-b8g5g"]
Nov 02 13:43:24.830420 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.830400    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:24.840244 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:24.840210    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0be5eb_c95f_462e_ab68_3e4dffaae950.slice/crio-cf080bc497163e6df396b87908e151ad84cd79af66f1362838e248b9ee49c57a WatchSource:0}: Error finding container cf080bc497163e6df396b87908e151ad84cd79af66f1362838e248b9ee49c57a: Status 404 returned error can't find the container with id cf080bc497163e6df396b87908e151ad84cd79af66f1362838e248b9ee49c57a
Nov 02 13:43:24.890409 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.890384    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 13:43:24.890539 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.890526    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mtv77\""
Nov 02 13:43:24.955748 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.955722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.955841 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.955757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.955904 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.955884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.955954 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.955930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.956002 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.955975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.956056 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.956041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.956150 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.956133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:24.956208 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:24.956158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057356 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057440 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057440 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057440 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057440 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057658 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057658 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057658 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.057843 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.057815    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.058338 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.058192    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.058338 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.058325    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.058484 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.058337    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.058484 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.058375    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.059995 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.059979    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.060056 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.060024    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.070324 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.070303    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.096329 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.096310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:25.227134 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.227099    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tczrv" event={"ID":"ae0be5eb-c95f-462e-ab68-3e4dffaae950","Type":"ContainerStarted","Data":"cf080bc497163e6df396b87908e151ad84cd79af66f1362838e248b9ee49c57a"}
Nov 02 13:43:25.375378 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.375355    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd7f9986f-b8g5g"]
Nov 02 13:43:25.376853 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:25.376816    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b7284f_c8bf_4bd4_8b71_16b297e9d2e8.slice/crio-4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383 WatchSource:0}: Error finding container 4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383: Status 404 returned error can't find the container with id 4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383
Nov 02 13:43:25.688407 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.688334    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:43:25.693825 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.693799    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.698647 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.698623    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 13:43:25.698870 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.698855    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 13:43:25.698955 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.698921    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 13:43:25.699023 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.698976    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 13:43:25.699083 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.699075    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 13:43:25.699136 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.699116    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 13:43:25.699136 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.699121    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f2hpb\""
Nov 02 13:43:25.699229 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.699162    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 13:43:25.699229 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.699186    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 13:43:25.699320 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.699309    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 13:43:25.717633 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.717612    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:43:25.763095 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763238 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763238 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763238 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763238 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763433 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763433 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763379    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763529 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763529 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763529 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs222\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763704 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763704 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.763704 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.763634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.863930 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.863900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864074 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.863939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864074 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.863970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs222\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864074 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864074 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864267 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864607 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.864289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.864607 ip-10-0-128-62 kubenswrapper[2575]: E1102 13:43:25.864542    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle podName:0c9eb4eb-569f-49eb-ac33-e670af0472a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:26.36452252 +0000 UTC m=+54.997925318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9") : configmap references non-existent config key: ca-bundle.crt
Nov 02 13:43:25.865718 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.865685    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.866509 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.866325    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.867618 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.867576    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.867919 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.867857    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.869159 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.868990    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.869742 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.869721    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.869954 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.869933    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.870061 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.870039    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.870495 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.870471    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.871676 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.871632    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.872286 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.872267    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:25.882375 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:25.882354    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs222\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:26.238860 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.238777    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae0be5eb-c95f-462e-ab68-3e4dffaae950" containerID="5f2613a21d71943d2b17734bef0ab7482b2980c75923f6492f1bf77461940b2e" exitCode=0
Nov 02 13:43:26.238995 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.238869    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tczrv" event={"ID":"ae0be5eb-c95f-462e-ab68-3e4dffaae950","Type":"ContainerDied","Data":"5f2613a21d71943d2b17734bef0ab7482b2980c75923f6492f1bf77461940b2e"}
Nov 02 13:43:26.240882 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.240848    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7f9986f-b8g5g" event={"ID":"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8","Type":"ContainerStarted","Data":"4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383"}
Nov 02 13:43:26.368694 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.368667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:26.369406 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.369381    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:26.621777 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.621743    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:26.682602 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.682572    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55c454df6-wgrrz"]
Nov 02 13:43:26.688648 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.688629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.697235 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.696204    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x2jws\""
Nov 02 13:43:26.697437 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.697355    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 13:43:26.704065 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.704038    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55c454df6-wgrrz"]
Nov 02 13:43:26.772305 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-metrics-client-ca\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772421 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772500 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772554 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772527    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-tmp\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772645 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ms8\" (UniqueName: \"kubernetes.io/projected/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-kube-api-access-s2ms8\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772702 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772667    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772753 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772704    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772818 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.772891 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.772826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-grpc-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.795085 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.795041    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 13:43:26.795085 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.795059    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 13:43:26.795212 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.795154    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-nsff4rjr46lv\""
Nov 02 13:43:26.795212 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.795170    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 13:43:26.795212 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.795185    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 13:43:26.873662 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873581    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873662 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873805 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-grpc-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873805 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-metrics-client-ca\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873805 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873948 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873948 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-tmp\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873948 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ms8\" (UniqueName: \"kubernetes.io/projected/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-kube-api-access-s2ms8\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.873948 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.873908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.874968 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.874475    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-tmp\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.875142 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.875119    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-metrics-client-ca\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.876756 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.876736    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.876952 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.876928    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.877033 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.877014    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.877324 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.877308    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.878461 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.878444    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-grpc-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.879448 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.879428    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.888121 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.888099    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ms8\" (UniqueName: \"kubernetes.io/projected/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-kube-api-access-s2ms8\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:26.923313 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:26.923287    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:43:26.927299 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:26.927278    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9eb4eb_569f_49eb_ac33_e670af0472a9.slice/crio-2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06 WatchSource:0}: Error finding container 2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06: Status 404 returned error can't find the container with id 2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06
Nov 02 13:43:27.018734 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:27.018705    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:27.246463 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:27.246377    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tczrv" event={"ID":"ae0be5eb-c95f-462e-ab68-3e4dffaae950","Type":"ContainerStarted","Data":"9fadacd47a1d229946ef901756e25a1487ecbf68152b7193cc22413873352ad9"}
Nov 02 13:43:27.246463 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:27.246440    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tczrv" event={"ID":"ae0be5eb-c95f-462e-ab68-3e4dffaae950","Type":"ContainerStarted","Data":"6bcde7f7406d3586aa5103b92777bda437374178b698213c0f0e640820764b90"}
Nov 02 13:43:27.247888 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:27.247865    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerStarted","Data":"2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06"}
Nov 02 13:43:27.270344 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:27.270294    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tczrv" podStartSLOduration=2.11398488 podStartE2EDuration="3.270282456s" totalImagesPullingTime="1.072129008s" totalInitContainerRuntime="84.168568ms" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:27.269291863 +0000 UTC m=+55.902694668" watchObservedRunningTime="2026-11-02 13:43:27.270282456 +0000 UTC m=+55.903685305"
Nov 02 13:43:27.287191 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:27.287172    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:27.312005 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:27.311982    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55c454df6-wgrrz"]
Nov 02 13:43:27.316244 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:27.316223    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfd9f5a_4325_4769_9b66_8ef4dc715b5c.slice/crio-b7c4d1b7b6aa0a3e4a0af1e0026791e59ebe7c0cf8ba35dee75ae3226881e44f WatchSource:0}: Error finding container b7c4d1b7b6aa0a3e4a0af1e0026791e59ebe7c0cf8ba35dee75ae3226881e44f: Status 404 returned error can't find the container with id b7c4d1b7b6aa0a3e4a0af1e0026791e59ebe7c0cf8ba35dee75ae3226881e44f
Nov 02 13:43:28.259531 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:28.259502    2575 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a" exitCode=0
Nov 02 13:43:28.259951 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:28.259617    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a"}
Nov 02 13:43:28.261723 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:28.261703    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" event={"ID":"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c","Type":"ContainerStarted","Data":"b7c4d1b7b6aa0a3e4a0af1e0026791e59ebe7c0cf8ba35dee75ae3226881e44f"}
Nov 02 13:43:29.222466 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.222435    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"]
Nov 02 13:43:29.225577 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.225539    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.244989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.244220    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 13:43:29.244989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.244274    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 13:43:29.244989 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.244389    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 13:43:29.245392 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.245369    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4c928\""
Nov 02 13:43:29.246103 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.246013    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-epoqrs26tg7nr\""
Nov 02 13:43:29.246211 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.246204    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 13:43:29.249456 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.247952    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"]
Nov 02 13:43:29.272428 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.272401    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7f9986f-b8g5g" event={"ID":"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8","Type":"ContainerStarted","Data":"8b3992990066f02c0a93407197f11fcf853eb899ba3ec186dc4d108812d5b4fd"}
Nov 02 13:43:29.299381 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.299347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.299517 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.299387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-metrics-server-audit-profiles\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.299635 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.299605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-client-certs\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.300668 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.299803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19f1ad48-874e-4463-b936-0704c67be0fd-audit-log\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.300668 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.299878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-client-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.300668 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.299908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-tls\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.300668 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.299930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ttd\" (UniqueName: \"kubernetes.io/projected/19f1ad48-874e-4463-b936-0704c67be0fd-kube-api-access-v6ttd\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.400500 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.400470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19f1ad48-874e-4463-b936-0704c67be0fd-audit-log\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.400625 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.400521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-client-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.400625 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.400546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-tls\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.400625 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.400598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ttd\" (UniqueName: \"kubernetes.io/projected/19f1ad48-874e-4463-b936-0704c67be0fd-kube-api-access-v6ttd\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.400746 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.400653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.400746 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.400688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-metrics-server-audit-profiles\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.400746 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.400722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-client-certs\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.401488 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.401457    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19f1ad48-874e-4463-b936-0704c67be0fd-audit-log\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.401899 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.401859    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.402681 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.402658    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-metrics-server-audit-profiles\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.404084 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.404063    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-client-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.404156 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.404100    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-client-certs\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.404156 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.404132    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-tls\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.414391 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.414353    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ttd\" (UniqueName: \"kubernetes.io/projected/19f1ad48-874e-4463-b936-0704c67be0fd-kube-api-access-v6ttd\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.539481 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.539453    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:29.830860 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.830818    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7dd7f9986f-b8g5g" podStartSLOduration=2.99846757 podStartE2EDuration="5.830803382s" totalImagesPullingTime="2.832335812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:29.308638429 +0000 UTC m=+57.942041234" watchObservedRunningTime="2026-11-02 13:43:29.830803382 +0000 UTC m=+58.464206186"
Nov 02 13:43:29.831174 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:29.831150    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"]
Nov 02 13:43:29.834910 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:29.834885    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f1ad48_874e_4463_b936_0704c67be0fd.slice/crio-62c45cd616f568bba1df46aa99e6c8b80efffa234cfa313b17ce0af2c7da2e6c WatchSource:0}: Error finding container 62c45cd616f568bba1df46aa99e6c8b80efffa234cfa313b17ce0af2c7da2e6c: Status 404 returned error can't find the container with id 62c45cd616f568bba1df46aa99e6c8b80efffa234cfa313b17ce0af2c7da2e6c
Nov 02 13:43:30.214656 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.214622    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:30.280045 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.279962    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" event={"ID":"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c","Type":"ContainerStarted","Data":"db47d7c6fef6ae0be29a30e812fefa95200bc8669ff5510e099726b11e37371c"}
Nov 02 13:43:30.280475 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.280055    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" event={"ID":"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c","Type":"ContainerStarted","Data":"1fc56ed62a174c58f56666af0340d66a560ccc6b93c28d8d60374f663604cf7d"}
Nov 02 13:43:30.280475 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.280098    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" event={"ID":"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c","Type":"ContainerStarted","Data":"529fb46abc6f9fe5aa22c3449d3174488b390420d41d5ffaa488dceca7e7384e"}
Nov 02 13:43:30.281384 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.281354    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c" event={"ID":"19f1ad48-874e-4463-b936-0704c67be0fd","Type":"ContainerStarted","Data":"62c45cd616f568bba1df46aa99e6c8b80efffa234cfa313b17ce0af2c7da2e6c"}
Nov 02 13:43:30.285074 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.285053    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerStarted","Data":"324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7"}
Nov 02 13:43:30.285184 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.285122    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerStarted","Data":"41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39"}
Nov 02 13:43:30.285184 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.285166    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerStarted","Data":"98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c"}
Nov 02 13:43:30.285289 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.285205    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerStarted","Data":"012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5"}
Nov 02 13:43:30.285289 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:30.285244    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerStarted","Data":"ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0"}
Nov 02 13:43:31.292306 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:31.292196    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerStarted","Data":"1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe"}
Nov 02 13:43:31.295586 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:31.295532    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" event={"ID":"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c","Type":"ContainerStarted","Data":"1b2dbd0084955f215695d7365202d5f2ba286219d6854cb88679d93caf292719"}
Nov 02 13:43:31.295703 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:31.295636    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" event={"ID":"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c","Type":"ContainerStarted","Data":"dfe55f5cc9be4e4dd2f98dadbbebf45224ffd836cac0101d34f5366b56ea0ef4"}
Nov 02 13:43:31.295703 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:31.295679    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" event={"ID":"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c","Type":"ContainerStarted","Data":"70ec7b4630f442214cec92dedd768cfe5be8aa942a0afa3db06dc2d73f23a993"}
Nov 02 13:43:31.296608 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:31.296586    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:31.331659 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:31.331614    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.08897545 podStartE2EDuration="6.331599453s" totalImagesPullingTime="3.224908077s" totalInitContainerRuntime="17.715926ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:31.330832157 +0000 UTC m=+59.964234962" watchObservedRunningTime="2026-11-02 13:43:31.331599453 +0000 UTC m=+59.965002257"
Nov 02 13:43:31.367117 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:31.367049    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" podStartSLOduration=2.5408335969999998 podStartE2EDuration="5.367029977s" totalImagesPullingTime="2.82619638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:31.366208489 +0000 UTC m=+59.999611343" watchObservedRunningTime="2026-11-02 13:43:31.367029977 +0000 UTC m=+60.000432783"
Nov 02 13:43:32.300594 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:32.300535    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c" event={"ID":"19f1ad48-874e-4463-b936-0704c67be0fd","Type":"ContainerStarted","Data":"4a0a915f28c17d22d78bbbc13beb1008a7513b7f6d463452e94c69e4ef9d00d7"}
Nov 02 13:43:32.335413 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:32.335368    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c" podStartSLOduration=1.895390556 podStartE2EDuration="3.335354215s" totalImagesPullingTime="1.439963659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:32.334336676 +0000 UTC m=+60.967739483" watchObservedRunningTime="2026-11-02 13:43:32.335354215 +0000 UTC m=+60.968757019"
Nov 02 13:43:33.364809 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:33.364776    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd7f9986f-b8g5g"]
Nov 02 13:43:35.097061 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:35.097020    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:37.685038 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.684996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:37.685495 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.685053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:37.685495 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.685077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:37.690249 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.690231    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:43:37.691678 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.691661    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:43:37.691762 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.691749    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:43:37.698285 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.698259    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:37.698379 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.698295    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:37.698920 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.698905    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:43:37.708416 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.708393    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:37.905632 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.905605    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m5gp5\""
Nov 02 13:43:37.912657 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.912629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:37.912895 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.912880    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qngrq\""
Nov 02 13:43:37.916125 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.916109    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzjbd\""
Nov 02 13:43:37.920207 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.920193    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:37.922978 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:37.922960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:38.196021 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:38.195815    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q6z7m"]
Nov 02 13:43:38.198094 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:38.198060    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c1e4ef_71b9_4a62_b841_a21453d83e59.slice/crio-b590fc8ab1d24597d7980fb62778395b323dd42ed02e9c3616187e4637230d22 WatchSource:0}: Error finding container b590fc8ab1d24597d7980fb62778395b323dd42ed02e9c3616187e4637230d22: Status 404 returned error can't find the container with id b590fc8ab1d24597d7980fb62778395b323dd42ed02e9c3616187e4637230d22
Nov 02 13:43:38.214837 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:38.214807    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7vld"]
Nov 02 13:43:38.217580 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:38.217528    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbsrr"]
Nov 02 13:43:38.218059 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:38.218034    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68685f74_14d9_444a_bbf1_9292f78d6a8e.slice/crio-1e68210f1659e4b50b4987638f2aec4a65a827114c41d35dee863a5c7f4ad9f0 WatchSource:0}: Error finding container 1e68210f1659e4b50b4987638f2aec4a65a827114c41d35dee863a5c7f4ad9f0: Status 404 returned error can't find the container with id 1e68210f1659e4b50b4987638f2aec4a65a827114c41d35dee863a5c7f4ad9f0
Nov 02 13:43:38.220516 ip-10-0-128-62 kubenswrapper[2575]: W1102 13:43:38.220498    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cccc0e_d0fc_4b34_badd_f4efbf8fd684.slice/crio-0592db798b9bcb260c4ebab756d705879765c84549991790d9b0116e61d73e35 WatchSource:0}: Error finding container 0592db798b9bcb260c4ebab756d705879765c84549991790d9b0116e61d73e35: Status 404 returned error can't find the container with id 0592db798b9bcb260c4ebab756d705879765c84549991790d9b0116e61d73e35
Nov 02 13:43:38.311101 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:38.311075    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:38.321829 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:38.321801    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbsrr" event={"ID":"91cccc0e-d0fc-4b34-badd-f4efbf8fd684","Type":"ContainerStarted","Data":"0592db798b9bcb260c4ebab756d705879765c84549991790d9b0116e61d73e35"}
Nov 02 13:43:38.322948 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:38.322926    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q6z7m" event={"ID":"71c1e4ef-71b9-4a62-b841-a21453d83e59","Type":"ContainerStarted","Data":"b590fc8ab1d24597d7980fb62778395b323dd42ed02e9c3616187e4637230d22"}
Nov 02 13:43:38.324051 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:38.324030    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7vld" event={"ID":"68685f74-14d9-444a-bbf1-9292f78d6a8e","Type":"ContainerStarted","Data":"1e68210f1659e4b50b4987638f2aec4a65a827114c41d35dee863a5c7f4ad9f0"}
Nov 02 13:43:40.335794 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:40.335754    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7vld" event={"ID":"68685f74-14d9-444a-bbf1-9292f78d6a8e","Type":"ContainerStarted","Data":"193c91b3b640252369b2e490738ef62e37dfb8b7a5d5708b9208364d96ffa83b"}
Nov 02 13:43:40.336214 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:40.335813    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7vld" event={"ID":"68685f74-14d9-444a-bbf1-9292f78d6a8e","Type":"ContainerStarted","Data":"dc3c187001629c1d8941b2c3f72f14f26a6b6cca4305f48c6a7d073eb35b8018"}
Nov 02 13:43:40.357263 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:40.357203    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l7vld" podStartSLOduration=67.981830202 podStartE2EDuration="1m9.357190514s" totalImagesPullingTime="1.375360312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:40.356264771 +0000 UTC m=+68.989667576" watchObservedRunningTime="2026-11-02 13:43:40.357190514 +0000 UTC m=+68.990593334"
Nov 02 13:43:41.671526 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:41.671480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jt476_70301f22-b14c-4362-a082-15309eafa41b/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:41.846956 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:41.846905    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jt476_70301f22-b14c-4362-a082-15309eafa41b/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:41.990786 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:41.990711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jt476_70301f22-b14c-4362-a082-15309eafa41b/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:42.347737 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:42.347704    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbsrr" event={"ID":"91cccc0e-d0fc-4b34-badd-f4efbf8fd684","Type":"ContainerStarted","Data":"21dcd6427fc2af7d7ab5f196b5cc747ccccf865b9cf6867d334467867a2d3c5a"}
Nov 02 13:43:42.347921 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:42.347837    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:42.368658 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:42.368598    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gbsrr" podStartSLOduration=67.906552759 podStartE2EDuration="1m11.368582655s" totalImagesPullingTime="3.462029896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:42.367396341 +0000 UTC m=+71.000799146" watchObservedRunningTime="2026-11-02 13:43:42.368582655 +0000 UTC m=+71.001985458"
Nov 02 13:43:43.353675 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:43.353596    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q6z7m" event={"ID":"71c1e4ef-71b9-4a62-b841-a21453d83e59","Type":"ContainerStarted","Data":"30965710d7b1e4d442e95438dda43b1eaeff954f7a50391719e35161c47ab190"}
Nov 02 13:43:43.357991 ip-10-0-128-62 kubenswrapper[2575]: I1102 13:43:43.357889    2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:43:43.358070 ip-10-0-128-62 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 13:43:43.370928 ip-10-0-128-62 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 13:43:43.371144 ip-10-0-128-62 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 13:43:43.371204 ip-10-0-128-62 systemd[1]: kubelet.service: Consumed 3.902s CPU time.
Nov 02 13:43:43.381179 ip-10-0-128-62 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:43:43.452662 ip-10-0-128-62 kubenswrapper[5572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:43:43.452662 ip-10-0-128-62 kubenswrapper[5572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:43:43.452662 ip-10-0-128-62 kubenswrapper[5572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:43:43.452662 ip-10-0-128-62 kubenswrapper[5572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:43:43.454938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454863    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.454938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454938    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454944    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454948    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454953    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454957    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454961    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454964    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454968    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454972    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454981    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454985    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454988    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454992    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.454996    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455000    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455003    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455007    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455012    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455016    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455020    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.455104 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455023    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455027    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455031    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455035    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455038    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455042    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455048    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455054    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455059    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455063    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455067    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455071    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455076    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455080    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455084    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455088    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455092    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455096    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455101    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455106    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.456008 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455112    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455117    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455122    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455127    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455131    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455135    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455139    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455144    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455148    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455152    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455157    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455161    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455165    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455169    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455173    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455177    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455181    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455185    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455189    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.456951 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455193    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455198    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455202    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455206    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455210    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455214    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455220    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455224    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455228    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455232    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455236    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455240    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455245    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455249    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455254    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455258    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455263    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455268    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455272    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.457829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455276    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455280    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455284    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455289    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455293    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455297    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455301    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455305    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455309    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455313    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455317    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455321    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455325    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455329    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455333    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455337    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455341    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455345    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455350    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455354    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.458705 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455358    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455362    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455366    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455372    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455376    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455381    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.455385    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456653    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456665    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456670    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456676    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456681    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456685    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456689    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456694    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456699    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456703    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456708    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456712    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.459648 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456716    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456721    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456725    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456729    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456733    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456737    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456741    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456746    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456750    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456754    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456758    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456762    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456767    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456770    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456775    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456778    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456782    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456787    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456792    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456801    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.460527 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456805    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456809    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456813    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456817    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456825    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456830    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456834    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456838    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456842    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456847    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456851    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456855    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456859    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456862    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456866    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456870    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456875    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456879    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456884    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.461313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456888    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456892    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456896    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456902    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456908    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456912    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456916    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456920    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456924    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456928    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456933    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456937    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456943    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456947    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456954    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456959    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456964    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456968    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456973    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.461817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456978    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456982    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456986    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456990    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456994    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.456998    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457002    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457006    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457010    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457015    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457019    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457023    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457027    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457031    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457035    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457039    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457044    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457048    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457052    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457056    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.462282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457060    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457065    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457069    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457074    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457078    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457085    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457090    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457094    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457098    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457102    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457106    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457110    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457115    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457119    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457123    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457127    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457986    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.457996    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458003    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458009    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.462829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458013    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458020    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458026    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458032    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458037    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458042    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458046    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458050    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458055    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458059    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458063    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458067    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458071    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458075    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458080    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458085    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458089    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458096    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458100    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.463359 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458105    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458109    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458113    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458118    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458125    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458132    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458136    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458141    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458145    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458149    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458153    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458157    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458161    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458165    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458169    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458174    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458178    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458182    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458186    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.463856 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458190    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458194    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458199    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458204    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458208    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458212    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458216    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458223    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458227    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458231    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458235    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458241    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458246    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458250    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458254    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458259    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458263    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458267    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458272    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458276    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.464313 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458280    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458285    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458289    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458293    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458297    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458302    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458306    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458310    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458314    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458318    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458322    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458326    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458330    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458334    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458338    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458343    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458347    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458351    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458355    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458361    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.464847 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458365    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458369    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458374    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458379    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458383    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458387    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458391    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458395    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458399    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458403    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458407    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458412    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458416    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458420    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458426    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458430    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458434    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458438    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458442    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.465325 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458446    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458450    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458454    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458458    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.458462    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458607    5572 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458619    5572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458628    5572 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458635    5572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458642    5572 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458647    5572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458655    5572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458663    5572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458668    5572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458674    5572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458679    5572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458685    5572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458690    5572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458695    5572 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458699    5572 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458704    5572 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458708    5572 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458713    5572 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458719    5572 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:43:43.465961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458724    5572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458729    5572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458734    5572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458740    5572 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458746    5572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458751    5572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458756    5572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458761    5572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458765    5572 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458770    5572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458774    5572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458780    5572 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458784    5572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458792    5572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458796    5572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458801    5572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458806    5572 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458810    5572 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458815    5572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458821    5572 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458829    5572 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458834    5572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458839    5572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458843    5572 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:43:43.466874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458850    5572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458854    5572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458859    5572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458863    5572 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458868    5572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458873    5572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458878    5572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458882    5572 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458887    5572 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458891    5572 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458896    5572 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458903    5572 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458907    5572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458912    5572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458917    5572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458922    5572 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458927    5572 flags.go:64] FLAG: --help="false"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458931    5572 flags.go:64] FLAG: --hostname-override="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458936    5572 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458941    5572 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458946    5572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458951    5572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458957    5572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:43:43.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458961    5572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458966    5572 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458970    5572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458975    5572 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458980    5572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458987    5572 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458991    5572 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.458996    5572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459000    5572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459005    5572 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459010    5572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459015    5572 flags.go:64] FLAG: --lock-file=""
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459019    5572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459024    5572 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459028    5572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459037    5572 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459041    5572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459046    5572 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459050    5572 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459055    5572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459060    5572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459065    5572 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459070    5572 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459077    5572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459082    5572 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:43:43.468034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459088    5572 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459092    5572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459097    5572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459102    5572 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459106    5572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459111    5572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459115    5572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459120    5572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459130    5572 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459134    5572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459139    5572 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459144    5572 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459150    5572 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459155    5572 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459159    5572 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459164    5572 flags.go:64] FLAG: --port="10250"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459169    5572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459173    5572 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00624b1e8dfb8c6fb"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459178    5572 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459183    5572 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459188    5572 flags.go:64] FLAG: --register-node="true"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459193    5572 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459198    5572 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459203    5572 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459207    5572 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459212    5572 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:43:43.468786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459218    5572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459223    5572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459227    5572 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459238    5572 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459243    5572 flags.go:64] FLAG: --runonce="false"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459248    5572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459253    5572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459258    5572 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459263    5572 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459267    5572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459272    5572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459277    5572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459282    5572 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459286    5572 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459290    5572 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459295    5572 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459299    5572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459304    5572 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459310    5572 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459315    5572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459326    5572 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459330    5572 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459335    5572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459342    5572 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459346    5572 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:43:43.469400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459351    5572 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459355    5572 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459360    5572 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459365    5572 flags.go:64] FLAG: --v="2"
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459371    5572 flags.go:64] FLAG: --version="false"
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459377    5572 flags.go:64] FLAG: --vmodule=""
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459383    5572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.459389    5572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459612    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459620    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459626    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459631    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459636    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459640    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459644    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459648    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459652    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459656    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459661    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459665    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459669    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.470167 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459674    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459678    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459683    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459687    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459691    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459696    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459702    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459706    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459711    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459715    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459719    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459723    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459727    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459731    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459736    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459740    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459744    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459749    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459755    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459761    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.470829 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459765    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459770    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459774    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459779    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459783    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459787    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459791    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459795    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459800    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459804    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459808    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459812    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459816    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459820    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459825    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459829    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459833    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459837    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459843    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.471395 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459847    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459851    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459856    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459860    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459864    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459868    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459872    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459877    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459881    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459885    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459889    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459893    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459897    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459901    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459906    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459910    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459914    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459920    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459925    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459929    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.471938 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459933    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459937    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459941    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459946    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459951    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459955    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459959    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459964    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459968    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459972    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459977    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459983    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459987    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459991    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.459995    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460001    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460005    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460009    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460013    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.472454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460017    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460021    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460025    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460030    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460036    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460043    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460049    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460054    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460059    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460063    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460067    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460072    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460076    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460080    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460084    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.460125    5572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:43:43.473028 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460186    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460192    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460197    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460202    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460206    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460210    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460214    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460219    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460223    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460229    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460233    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460239    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460244    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460248    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460253    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460257    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460261    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460265    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460269    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.473510 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460273    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460280    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460286    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460292    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460297    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460302    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460306    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460310    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460315    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460320    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460325    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460329    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460333    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460337    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460341    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460345    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460349    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460353    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460357    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.474019 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460361    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460365    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460369    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460375    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460379    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460384    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460389    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460393    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460397    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460401    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460405    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460409    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460413    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460419    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460423    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460427    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460431    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460435    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460439    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460443    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.474490 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460447    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460451    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460455    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460460    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460467    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460473    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460478    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460482    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460485    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460490    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460494    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460498    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460502    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460506    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460511    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460516    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460521    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460526    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460530    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.475126 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460534    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460538    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460541    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460545    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460549    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460553    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460575    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460580    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460584    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460588    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460592    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460596    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460600    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460604    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460609    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460613    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460617    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460621    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460626    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.475605 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460630    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460634    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460638    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460642    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460646    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460651    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460655    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460660    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460664    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.460670    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.464649    5572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.464662    5572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.464680    5572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.464684    5572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464738    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464743    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464746    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464749    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464752    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464756    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.476051 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464759    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464761    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464763    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464766    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464768    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464771    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464773    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464776    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464779    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464781    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464784    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464787    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464790    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464792    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464795    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464798    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464800    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464803    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464805    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464808    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.476551 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464810    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464812    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464815    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464817    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464820    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464822    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464824    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464827    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464829    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464832    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464835    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464837    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464840    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464842    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464844    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464847    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464849    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464852    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464854    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.477042 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464857    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464860    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464862    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464865    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464868    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464871    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464874    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464876    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464879    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464881    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464897    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464900    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464903    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464906    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464908    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464910    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464913    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464915    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464918    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464920    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.477492 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464923    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464927    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464931    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464934    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464937    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464940    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464943    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464945    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464948    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464950    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464953    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464955    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464957    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464960    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464962    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464972    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464975    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464980    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464982    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.478026 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464985    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464989    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464993    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.464997    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465000    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465002    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465005    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465007    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465010    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465012    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465015    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465018    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465020    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465023    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465025    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465028    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465030    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465033    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465035    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.478480 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465038    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465040    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465043    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.465070    5572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465227    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465232    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465235    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465237    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465240    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465242    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465245    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465247    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465250    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465253    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465256    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465258    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.478957 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465260    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465263    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465265    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465268    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465270    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465273    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465275    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465277    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465280    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465282    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465285    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465287    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465290    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465292    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465294    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465297    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465299    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465302    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465304    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.479341 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465306    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465309    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465311    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465317    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465320    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465322    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465325    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465328    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465331    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465333    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465336    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465339    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465341    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465344    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465346    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465349    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465351    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465354    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465356    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465359    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.479813 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465361    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465364    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465366    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465369    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465371    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465374    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465376    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465378    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465381    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465383    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465386    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465388    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465390    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465393    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465395    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465399    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465402    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465405    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465407    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.480283 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465409    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465412    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465414    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465417    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465420    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465434    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465438    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465440    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465443    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465445    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465447    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465450    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465452    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465455    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465457    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465460    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465462    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465465    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465467    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465469    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.480852 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465472    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465474    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465477    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465479    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465481    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465484    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465486    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465489    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465492    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465496    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465500    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465504    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465508    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465511    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465514    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.481613 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.465517    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.465540    5572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466246    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466263    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466268    5572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466274    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466278    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466282    5572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466286    5572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466290    5572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466298    5572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466311    5572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466318    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466323    5572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466328    5572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:43.482023 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466333    5572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466337    5572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466341    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466346    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466350    5572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466354    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466358    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466362    5572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466366    5572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466375    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466380    5572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466385    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466390    5572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466394    5572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466398    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466403    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466407    5572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466411    5572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466415    5572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:43.482570 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466420    5572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466425    5572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466434    5572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466438    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466443    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466447    5572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466452    5572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466455    5572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466459    5572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466463    5572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466469    5572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466474    5572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466478    5572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466482    5572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466491    5572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466495    5572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466499    5572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466503    5572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466507    5572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:43.483021 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466511    5572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466516    5572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466520    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466525    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466529    5572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466536    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466540    5572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466550    5572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466554    5572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466579    5572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466583    5572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466588    5572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466592    5572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466596    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466600    5572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466605    5572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466609    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466613    5572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466617    5572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466621    5572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:43.483640 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466629    5572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466633    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466637    5572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466641    5572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466678    5572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466682    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466685    5572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466687    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466690    5572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466692    5572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466695    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466697    5572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466700    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466703    5572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466706    5572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466708    5572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466711    5572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466738    5572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466741    5572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466832    5572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:43.484149 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466836    5572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466839    5572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466841    5572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466844    5572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466847    5572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466849    5572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466852    5572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466859    5572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466861    5572 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466864    5572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466867    5572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466869    5572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466872    5572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466875    5572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:43:43.466877    5572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.466949    5572 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.467880    5572 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.467928    5572 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.468090    5572 server.go:1014] "Starting client certificate rotation"
Nov 02 13:43:43.484941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.468233    5572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.468285    5572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:37:31 +0000 UTC" deadline="2028-04-23 07:51:56.953485007 +0000 UTC"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.468324    5572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12906h8m13.485164595s"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.468502    5572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.469583    5572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.470894    5572 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.472623    5572 log.go:25] "Validated CRI v1 image API"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.473396    5572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:43:43.485724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.475366    5572 fs.go:135] Filesystem UUIDs: map[33daa21b-7504-4a21-af30-ebd5d61aeca2:/dev/nvme0n1p4 5b332bc9-d2c7-4ad4-a14a-982a1a07b270:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 02 13:43:43.486094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.475380    5572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0390309ce3f675d966a964da3efa3a78c1da668837cc046ea37bdd874b969d9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0390309ce3f675d966a964da3efa3a78c1da668837cc046ea37bdd874b969d9a/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0592db798b9bcb260c4ebab756d705879765c84549991790d9b0116e61d73e35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0592db798b9bcb260c4ebab756d705879765c84549991790d9b0116e61d73e35/userdata/shm major:0 minor:405 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dc1303df8c6a2a326a300c97835a1ede77ada40caf150cd1fe5b8c68cb4deef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dc1303df8c6a2a326a300c97835a1ede77ada40caf150cd1fe5b8c68cb4deef/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15f87d486fc614b1378b929466f0e8fc74494592f7d06bdeccd942824cb3be1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15f87d486fc614b1378b929466f0e8fc74494592f7d06bdeccd942824cb3be1c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dae972d8b96164201c2d90321d585deca175e8ae96015e1f04603dcdfaf0eb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dae972d8b96164201c2d90321d585deca175e8ae96015e1f04603dcdfaf0eb1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e68210f1659e4b50b4987638f2aec4a65a827114c41d35dee863a5c7f4ad9f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e68210f1659e4b50b4987638f2aec4a65a827114c41d35dee863a5c7f4ad9f0/userdata/shm major:0 minor:404 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/253ef5f2b34461ff99fab354c090190b292a59d5c554408e35aa4ba5bc66f990/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/253ef5f2b34461ff99fab354c090190b292a59d5c554408e35aa4ba5bc66f990/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a3dea0116d7928578aa74c6ef9a427180ba71a8d4716145f9cf741641d1137c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a3dea0116d7928578aa74c6ef9a427180ba71a8d4716145f9cf741641d1137c/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62a68096776c43d66201f8f2b671e5eb981807dba25263666710e31a1bc73b22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62a68096776c43d66201f8f2b671e5eb981807dba25263666710e31a1bc73b22/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62c45cd616f568bba1df46aa99e6c8b80efffa234cfa313b17ce0af2c7da2e6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62c45cd616f568bba1df46aa99e6c8b80efffa234cfa313b17ce0af2c7da2e6c/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89fb3ac01094f442a7e0a52f0b7853253147c5772fb5159baf1c84c8c20fe55c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89fb3ac01094f442a7e0a52f0b7853253147c5772fb5159baf1c84c8c20fe55c/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f13ce4dc3a40098afa1318defbbc3bd3d77288494ed65e6a116a5465a4105c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f13ce4dc3a40098afa1318defbbc3bd3d77288494ed65e6a116a5465a4105c1/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b590fc8ab1d24597d7980fb62778395b323dd42ed02e9c3616187e4637230d22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b590fc8ab1d24597d7980fb62778395b323dd42ed02e9c3616187e4637230d22/userdata/shm major:0 minor:402 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7c4d1b7b6aa0a3e4a0af1e0026791e59ebe7c0cf8ba35dee75ae3226881e44f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7c4d1b7b6aa0a3e4a0af1e0026791e59ebe7c0cf8ba35dee75ae3226881e44f/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7c0fc1931916b9511c1ce75cfbf7b26dacd2444935d229f233392d2b9ec5e86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7c0fc1931916b9511c1ce75cfbf7b26dacd2444935d229f233392d2b9ec5e86/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf080bc497163e6df396b87908e151ad84cd79af66f1362838e248b9ee49c57a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf080bc497163e6df396b87908e151ad84cd79af66f1362838e248b9ee49c57a/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e70b3b6d1ee0b55591a26e3b5da81fe703c2f5d1d542a14606b327bb09be8f19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e70b3b6d1ee0b55591a26e3b5da81fe703c2f5d1d542a14606b327bb09be8f19/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f02f0af1d313803870af6eeeeb0599183b0f3d8c805015a1cd2b353a76e7f625/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f02f0af1d313803870af6eeeeb0599183b0f3d8c805015a1cd2b353a76e7f625/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/alertmanager/9 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~empty-dir/config-out major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~projected/kube-api-access-zs222:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~projected/kube-api-access-zs222 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~projected/tls-assets major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/config-volume major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/web-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~projected/kube-api-access-v6ttd:{mountpoint:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~projected/kube-api-access-v6ttd major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21c18bda-36de-4e16-9ea6-952ace591a1f/volumes/kubernetes.io~projected/kube-api-access-6nrfg:{mountpoint:/var/lib/kubelet/pods/21c18bda-36de-4e16-9ea6-952ace591a1f/volumes/kubernetes.io~projected/kube-api-access-6nrfg major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21c18bda-36de-4e16-9ea6-952ace591a1f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/21c18bda-36de-4e16-9ea6-952ace591a1f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volumes/kubernetes.io~projected/kube-api-access-hnj9d:{mountpoint:/var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volumes/kubernetes.io~projected/kube-api-access-hnj9d major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/582ae8d7-d1b6-4066-a863-a14a5f2eb45f/volumes/kubernetes.io~projected/kube-api-access-x2b48:{mountpoint:/var/lib/kubelet/pods/582ae8d7-d1b6-4066-a863-a14a5f2eb45f/volumes/kubernetes.io~projected/kube-api-access-x2b48 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/582ae8d7-d1b6-4066-a863-a14a5f2eb45f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/582ae8d7-d1b6-4066-a863-a14a5f2eb45f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea/volumes/kubernetes.io~projected/kube-api-access-snsd7:{mountpoint:/var/lib/kubelet/pods/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea/volumes/kubernetes.io~projected/kube-api-access-snsd7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~projected/kube-api-access-w2hgh:{mountpoint:/var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~projected/kube-api-access-w2hgh major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68685f74-14d9-444a-bbf1-9292f78d6a8e/volumes/kubernetes.io~projected/kube-api-access-24hl9:{mountpoint:/var/lib/kubelet/pods/68685f74-14d9-444a-bbf1-9292f78d6a8e/volumes/kubernetes.io~projected/kube-api-access-24hl9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68685f74-14d9-444a-bbf1-9292f78d6a8e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/68685f74-14d9-444a-bbf1-9292f78d6a8e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70301f22-b14c-4362-a082-15309eafa41b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/70301f22-b14c-4362-a082-15309eafa41b/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c1e4ef-71b9-4a62-b841-a21453d83e59/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/71c1e4ef-71b9-4a62-b841-a21453d83e59/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73189526-2e64-4de5-a2a9-06597d293b6b/volumes/kubernetes.io~projected/kube-api-access-hnxqj:{mountpoint:/var/lib/kubelet/pods/73189526-2e64-4de5-a2a9-06597d293b6b/volumes/kubernetes.io~projected/kube-api-access-hnxqj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b652b34-c239-4b27-ada2-ec7b39d17539/volumes/kubernetes.io~projected/kube-api-access-9nltd:{mountpoint:/var/lib/kubelet/pods/7b652b34-c239-4b27-ada2-ec7b39d17539/volumes/kubernetes.io~projected/kube-api-access-9nltd major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~projected/kube-api-access-5wjzw:{mountpoint:/var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~projected/kube-api-access-5wjzw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~projected/kube-api-access-s2ms8:{mountpoint:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~projected/kube-api-access-s2ms8 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91cccc0e-d0fc-4b34-badd-f4efbf8fd684/volumes/kubernetes.io~projected/kube-api-access-vjbql:{mountpoint:/var/lib/kubelet/pods/91cccc0e-d0fc-4b34-badd-f4efbf8fd684/volumes/kubernetes.io~projected/kube-api-access-vjbql major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c6359e8-e292-457f-b680-f6b44bee7e79/volumes/kubernetes.io~projected/kube-api-access-rgzxj:{mountpoint:/var/lib/kubelet/pods/9c6359e8-e292-457f-b680-f6b44bee7e79/volumes/kubernetes.io~projected/kube-api-access-rgzxj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~projected/kube-api-access-9k9fw:{mountpoint:/var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~projected/kube-api-access-9k9fw major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~projected/kube-api-access-tmjdj:{mountpoint:/var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~projected/kube-api-access-tmjdj major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b672addc-8b6e-4983-bc93-17ff035c88bf/volumes/kubernetes.io~projected/kube-api-access-kkvvp:{mountpoint:/var/lib/kubelet/pods/b672addc-8b6e-4983-bc93-17ff035c88bf/volumes/kubernetes.io~projected/kube-api-access-kkvvp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca506817-08d9-420e-b25d-45dc0de360c4/volumes/kubernetes.io~projected/kube-api-access-27t78:{mountpoint:/var/lib/kubelet/pods/ca506817-08d9-420e-b25d-45dc0de360c4/volumes/kubernetes.io~projected/kube-api-access-27t78 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf3f6767-bfe3-4c84-b337-e17679488a51/volumes/kubernetes.io~projected/kube-api-access-97q6p:{mountpoint:/var/lib/kubelet/pods/cf3f6767-bfe3-4c84-b337-e17679488a51/volumes/kubernetes.io~projected/kube-api-access-97q6p major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf3f6767-bfe3-4c84-b337-e17679488a51/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cf3f6767-bfe3-4c84-b337-e17679488a51/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c077d789f7c824c8af04e5c13c08627b0f14695ff20d4b0be1c05ee0328026b4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5c49ea037f6411981e734066725ae7ba09cd29b9a0877e4fc6e4d29107519184/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e4ba56930ab8fe02a50147790787e4b25cb7675c9665b3cb08395731bd73e6d1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2d49e08895a80c0107f314141f4be466386f37422eb852d2fe09e73df8d7d8c2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9c79b9a56614110565c75434122affd49f63dea3a93e5a04de0717a8f6294a4b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5543e2ce427df4731fda54e3d8f4810e0b50bb73336fad650b520da1019ced51/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/10c1204a492bf2a0a3249bfc98081ff090520418feacc631d00eaad84a0a39c7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a655db6af5ae6654a0ba3444ba77b942c3e26a89a6ca15b41422cdb8a6b32cad/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3d97bf9aa1a9c496d665aa5fb5c39e38f7f73a182498d9fbecfddece722f5246/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/e77b9e0594df939cbbdf96a01feb664039f0313cb0154458f9f277af40705855/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/06bb0933e394a8917bb81e4cc42cb748495d3b06af395cd13a9e854c464cf7d6/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/34b4a823a34770049727ea9740c07647efbc08ff03c27b247c8fe0d9e39a575e/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1427d30173121fe0fd4c9e16bcee306b0844f67767754f85f7a15c7ba9c6ad66/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/1089b5892d43ee3192beb01c723a10dcdf3e5133f1e9d304786fa924a0c83a00/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/bb7972b91a57d4ebe040a8d552ff74e0934ebe1d8bf60afeb0925624f200dc8c/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/23eac6c0341c020e3e8cc87123c7074b440cb0f7c7e9771827705fbe78c175ec/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/216cc7cb2989ba04f4780988237d4b3618a8d93e7be4efcb0d45e8ef9dd8a737/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fbbf2e3a13819815bd8e78917f58a1e91169a3f9c4cd86be057a9d624a7e97a1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a1117a0f2d34465f8cb5c91e0d5292d48a95fbab9c2666ca349fffde9064cff7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fabd0b0e21086b97d10a7683311d60b80033de8b0f7960b863e9c0f55ee05065/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e6c4a97f44c23f9a6145473ac0cfe91be9ee20427a655d4577536fa37feb946e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/709ed49531197bb7745a007026110ac8a7480bd3eebd533de311e5821ff78c1c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/a78dfe87263f6e053143a6d36625a06ad3a752c11df9dc57774041d72366e5a2/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/4b031c2e68cb300e3e19c3e08faf1eaf3fa94cecd266fa6772930f61f3457109/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d7730eb2efd968c6ccd33e118dcfbfbc2541f87d8605dc22e5c750bb3da64eee/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/419f761d3d8825cb8f29de504265a0782de3738f5582664c3c7b2fdaf075de3a/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/0c0e05831af5139fa6bbeb77a7ec2f15852f41396823a1c4685284896a1442f6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/6f855078ffbd3e1ca75530956b5158beaf0a3d061ab9667e6bf9b4d7dcaf8f85/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/e1c00d511bf8a111772fa850644c2096bd3cfa9b6639b75113defe560dcd3bcd/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/1b9078518a4d15da53b7a27660d7e1152670f35ebed247b019ea78407535f9d1/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/808c80730b913dc4964fe956bfb75f17dbca7281738ef486db6ff1e8efa547da/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/908d9f4d5b185d8e4250a04c2f532398b042f0c20372e24260440fb7a359f109/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/648bf3a5bb2f4cf15eb402da8280db1c776597dd3d7803405bdd873d459c99eb/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/442ded96ce8262cc816fc0a7ec685dc5f06422e9d4d1ecaedc1c68c347da19be/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/0e0f9022b33243028dc38044429768213cb2bd5fc6024f49f1a9761153fec867/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/243961628dd76238615a0e3637f6762a095a735c4b40e926151047f6043d0841/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/73310c79fa5cda987a77c0064df1604ae9087a05ea729b8321afc2404335b4a2/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/e55555138f81557a71461787b4a79d058f46c0714c9516f337b661c3bdfdc7a6/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/2d03630636c8c9e1161d6f166bacac731fab6bf5992dd83477b686064e5c14c3/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/27627ac93c70b1562505a52a5aeed606f1c69aa57a08e3e672d26b4848834615/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/003a0ad0cd5e484e5de8976f1bc0f01a077309833e9c2c9f6532013e4052738d/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/271ec03e399eeca45db1023435ac48896a2c63bb22e458a6c06f8fbb50f6c5f6/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/f47ae162f3b1e3014bb51747f1af524c96876a28ef705c94db52936331522b07/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/d494f6010db68b6a11551883f5018822253f7cdf6fe562f2d5ff64bdb509ffc6/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/5411fcb902195ffec0b716bf0fcd423ecaadd3857b69cb7a645e64e78ba3c66c/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/7e24114f0c68732bdf102ad4ae5e2c11ce503776ff5e8162c367d582304608c9/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/47a000eaa6ff9090e36ec9ec999cbb2f96f3f1c915bb20e25bf5488cb07c4798/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/b683e75be6aa053853c0496b571d857ba02882cc945e6e122fcf5cc02dd496d2/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/3b926556d8ebd48c7668a0bb69cb39008a7885aea287fdd4645bd82eca1c5b94/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/82b8c7f5fc0fa575c02d799f8f53b75bb40067d314681668955f634a833323b1/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/abbe6f44dc82f91e7e688ba0f7173598529b449ea2bfb604a2698970af9b60b9/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/7ed0fd8f7726c59c7dba178fbd4bd72e4f808115ce29cc7e07a81dc7743592ee/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/353d9287e94460c6aed72ad26519735693fe11abbbd45e3452984262199566dd/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/eaef608b615c7c924b4971284fe0e7f09fcc41fbaf91234ef8e1fd68f9446b61/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/61ca6d9cda311587cdbaccd06e841bac7d43ddc06c436ea1fe53db928235b6bf/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/bf26831692ee3c71cd61928c633215c2aa4cdf3f8b134aa4031ce2af5eb9ed85/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/f2783341be3df990078926de6f77edee6f637e96a819972584c4bc939aeef830/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/71e74f60760fc5789bb14a508c427e9f6fb00d6952ae9168903d239fb2786b97/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/15f5cb6390057d8b4ec7beff27d32bf703024bf533c58cf69fee698a8eb913f6/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/b886e4f1dd79b4082f620bbd85ddccaac1b7495a15b0de76fce123e53155c503/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/a4b72c5c9d4fb1b3f97a286a28e1c288f5cd18ac27c061c4f4ed427123dc806b/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/aa6d883f13d478d1b30d76505fa22fa8bf30cf10a056cf6867b6cd822fca20d9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7a0b8e15f2017cca18d85e8bc142e588a3d472aeed88b393c7a6b0f60d6eb552/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f682a6a07b83c25d9844ab4038f878f17050b1ffb5ddfad66f2b9ff19afcb70b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/32f9dbba61d99735bbae4df2302f30969374f4986eab80747c521a88a2a3f1f3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/a2022c7a2df24527dbc9a11e3dc5840b19c6b77d95caf986516d619186f2a164/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2a0000303c5c9554d91197ab6ed6570a74d7009324b047b13dc7b8a713093956/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8a01bb03cd253d1c5dffdde06a235c78b7f0b926ce389e9ef4d4294a405544f1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0071de7dcdcf358b3dd6e176e32e1cd4eb48522763604e83cc0c6b9769e7e364/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fd14faa763df143034a8c5734ba80732a42ba2e23c0cebffd1fedb678a1c8dc9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2722c1b33da8d77ba564199a33b1b89c273cd7a7a1c4fe2e228d92922363642c/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.483949    5572 manager.go:217] Machine: {Timestamp:2026-11-02 13:43:43.483483066 +0000 UTC m=+0.061571925 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3131798 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0f6d90ea04eb67a993d917877980 SystemUUID:ec2d0f6d-90ea-04eb-67a9-93d917877980 BootID:43ca23b7-b60d-4dd8-b49f-97c909c96d24 Filesystems:[{Device:/var/lib/kubelet/pods/9c6359e8-e292-457f-b680-f6b44bee7e79/volumes/kubernetes.io~projected/kube-api-access-rgzxj DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:290 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:304 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68685f74-14d9-444a-bbf1-9292f78d6a8e/volumes/kubernetes.io~projected/kube-api-access-24hl9 DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dae972d8b96164201c2d90321d585deca175e8ae96015e1f04603dcdfaf0eb1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68685f74-14d9-444a-bbf1-9292f78d6a8e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:395 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0592db798b9bcb260c4ebab756d705879765c84549991790d9b0116e61d73e35/userdata/shm DeviceMajor:0 DeviceMinor:405 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a3dea0116d7928578aa74c6ef9a427180ba71a8d4716145f9cf741641d1137c/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:283 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~projected/kube-api-access-s2ms8 DeviceMajor:0 DeviceMinor:310 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e70b3b6d1ee0b55591a26e3b5da81fe703c2f5d1d542a14606b327bb09be8f19/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:306 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:327 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71c1e4ef-71b9-4a62-b841-a21453d83e59/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:400 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21c18bda-36de-4e16-9ea6-952ace591a1f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~projected/kube-api-access-9k9fw DeviceMajor:0 DeviceMinor:252 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dc1303df8c6a2a326a300c97835a1ede77ada40caf150cd1fe5b8c68cb4deef/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:307 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e68210f1659e4b50b4987638f2aec4a65a827114c41d35dee863a5c7f4ad9f0/userdata/shm DeviceMajor:0 DeviceMinor:404 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b672addc-8b6e-4983-bc93-17ff035c88bf/volumes/kubernetes.io~projected/kube-api-access-kkvvp DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/253ef5f2b34461ff99fab354c090190b292a59d5c554408e35aa4ba5bc66f990/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~projected/kube-api-access-5wjzw DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca506817-08d9-420e-b25d-45dc0de360c4/volumes/kubernetes.io~projected/kube-api-access-27t78 DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:308 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89fb3ac01094f442a7e0a52f0b7853253147c5772fb5159baf1c84c8c20fe55c/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/73189526-2e64-4de5-a2a9-06597d293b6b/volumes/kubernetes.io~projected/kube-api-access-hnxqj DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf3f6767-bfe3-4c84-b337-e17679488a51/volumes/kubernetes.io~projected/kube-api-access-97q6p DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:290 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7c0fc1931916b9511c1ce75cfbf7b26dacd2444935d229f233392d2b9ec5e86/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/582ae8d7-d1b6-4066-a863-a14a5f2eb45f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/91cccc0e-d0fc-4b34-badd-f4efbf8fd684/volumes/kubernetes.io~projected/kube-api-access-vjbql DeviceMajor:0 DeviceMinor:401 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volumes/kubernetes.io~projected/kube-api-access-hnj9d DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62a68096776c43d66201f8f2b671e5eb981807dba25263666710e31a1bc73b22/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~projected/kube-api-access-w2hgh DeviceMajor:0 DeviceMinor:277 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:309 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:326 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf080bc497163e6df396b87908e151ad84cd79af66f1362838e248b9ee49c57a/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:285 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/21c18bda-36de-4e16-9ea6-952ace591a1f/volumes/kubernetes.io~projected/kube-api-access-6nrfg DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f13ce4dc3a40098afa1318defbbc3bd3d77288494ed65e6a116a5465a4105c1/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b376c48-407e-4913-8b07-a0eb3fcc67a0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea/volumes/kubernetes.io~projected/kube-api-access-snsd7 DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/582ae8d7-d1b6-4066-a863-a14a5f2eb45f/volumes/kubernetes.io~projected/kube-api-access-x2b48 DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/aa65ead1-60e6-42c4-bc86-ef6df717b751/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:246 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:305 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f02f0af1d313803870af6eeeeb0599183b0f3d8c805015a1cd2b353a76e7f625/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:289 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~projected/kube-api-access-zs222 DeviceMajor:0 DeviceMinor:291 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70301f22-b14c-4362-a082-15309eafa41b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0390309ce3f675d966a964da3efa3a78c1da668837cc046ea37bdd874b969d9a/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:276 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:282 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15f87d486fc614b1378b929466f0e8fc74494592f7d06bdeccd942824cb3be1c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:284 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62c45cd616f568bba1df46aa99e6c8b80efffa234cfa313b17ce0af2c7da2e6c/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b590fc8ab1d24597d7980fb62778395b323dd42ed02e9c3616187e4637230d22/userdata/shm DeviceMajor:0 DeviceMinor:402 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:325 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b652b34-c239-4b27-ada2-ec7b39d17539/volumes/kubernetes.io~projected/kube-api-access-9nltd DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~projected/kube-api-access-tmjdj DeviceMajor:0 DeviceMinor:270 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7c4d1b7b6aa0a3e4a0af1e0026791e59ebe7c0cf8ba35dee75ae3226881e44f/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/33b41081-1ac8-414a-9911-450a0b7b7d7f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf3f6767-bfe3-4c84-b337-e17679488a51/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae0be5eb-c95f-462e-ab68-3e4dffaae950/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/19f1ad48-874e-4463-b936-0704c67be0fd/volumes/kubernetes.io~projected/kube-api-access-v6ttd DeviceMajor:0 DeviceMinor:328 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0592db798b9bcb2 MacAddress:6a:54:55:c9:89:aa Speed:10000 Mtu:8901} {Name:0dc1303df8c6a2a MacAddress:3e:f0:4e:f4:e4:df Speed:10000 Mtu:8901} {Name:1e68210f1659e4b MacAddress:de:9c:70:fc:cf:e1 Speed:10000 Mtu:8901} {Name:253ef5f2b34461f MacAddress:f6:a2:e8:07:f7:0c Speed:10000 Mtu:8901} {Name:2e94e799b19519e MacAddress:76:92:8e:f8:62:cd Speed:10000 Mtu:8901} {Name:4898619cf29b852 MacAddress:7a:0b:79:40:9e:a0 Speed:10000 Mtu:8901} {Name:62c45cd616f568b MacAddress:3a:ee:a9:a4:bb:b5 Speed:10000 Mtu:8901} {Name:89fb3ac01094f44 MacAddress:a2:51:0f:86:d8:6d Speed:10000 Mtu:8901} {Name:b590fc8ab1d2459 MacAddress:a6:ea:db:9a:43:7e Speed:10000 Mtu:8901} {Name:b7c4d1b7b6aa0a3 MacAddress:32:d3:4e:a3:82:1a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:84:d0:fd Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:61:80:0b:22:0e Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:84:d0:fd Speed:0 Mtu:9001} {Name:f02f0af1d313803 MacAddress:0e:72:87:df:af:b9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:e6:5f:3a:4e:dd:08 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:26:5c:a9:bc:ba:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484309    5572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484347    5572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484699    5572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484731    5572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-62.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484909    5572 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484921    5572 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484938    5572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.484963    5572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485060    5572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485251    5572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485392    5572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485441    5572 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485456    5572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485469    5572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485481    5572 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.485492    5572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.486342    5572 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:43:43.486598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.486355    5572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:43:43.487300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.486877    5572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:43:43.489262 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489248    5572 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:43:43.489262 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489262    5572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:43:43.489361 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489352    5572 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 13:43:43.489529 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489520    5572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:43:43.489574 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489548    5572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:43:43.489657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489648    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:43:43.489698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489663    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:43:43.489698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489670    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:43:43.489698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489675    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:43:43.489698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489681    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:43:43.489698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489687    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:43:43.489698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489700    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:43:43.490003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489706    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:43:43.490003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489713    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:43:43.490003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489718    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:43:43.490003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489726    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:43:43.490003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489734    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:43:43.490003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489757    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:43:43.490003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.489763    5572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:43:43.493090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.493077    5572 server.go:1263] "Started kubelet"
Nov 02 13:43:43.493162 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.493142    5572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:43:43.495607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.493224    5572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:43:43.495607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.493307    5572 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:43:43.495607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.493631    5572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:43:43.495607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.494619    5572 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:43:43.493612 ip-10-0-128-62 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:43:43.501714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.501423    5572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:43:43.501714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.501452    5572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:43:43.501714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.501630    5572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:37:33 +0000 UTC" deadline="2028-05-09 22:11:10.941804645 +0000 UTC"
Nov 02 13:43:43.501894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.501725    5572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13304h27m27.440086252s"
Nov 02 13:43:43.501995 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:43.501974    5572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-62.us-east-2.compute.internal\" not found"
Nov 02 13:43:43.502285 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.502271    5572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:43:43.502343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.502271    5572 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:43:43.502343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.502297    5572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:43:43.504628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.504542    5572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:43:43.504628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.504583    5572 factory.go:55] Registering systemd factory
Nov 02 13:43:43.504628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.504593    5572 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:43:43.504859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.504842    5572 factory.go:153] Registering CRI-O factory
Nov 02 13:43:43.504923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.504861    5572 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:43:43.504923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.504877    5572 factory.go:103] Registering Raw factory
Nov 02 13:43:43.504923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.504889    5572 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:43:43.505320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505303    5572 manager.go:319] Starting recovery of all containers
Nov 02 13:43:43.505935 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505907    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f1ad48-874e-4463-b936-0704c67be0fd" volumeName="kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 02 13:43:43.506013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505935    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa65ead1-60e6-42c4-bc86-ef6df717b751" volumeName="kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 13:43:43.506013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505948    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets" seLinuxMountContext=""
Nov 02 13:43:43.506013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505959    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 13:43:43.506013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505971    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 13:43:43.506013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505981    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 02 13:43:43.506013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.505993    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21c18bda-36de-4e16-9ea6-952ace591a1f" volumeName="kubernetes.io/projected/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-api-access-6nrfg" seLinuxMountContext=""
Nov 02 13:43:43.506013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506004    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/empty-dir/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-tmp" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506018    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506029    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0be5eb-c95f-462e-ab68-3e4dffaae950" volumeName="kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506040    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f1ad48-874e-4463-b936-0704c67be0fd" volumeName="kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-tls" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506052    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506064    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506077    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf3f6767-bfe3-4c84-b337-e17679488a51" volumeName="kubernetes.io/projected/cf3f6767-bfe3-4c84-b337-e17679488a51-kube-api-access-97q6p" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506091    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506104    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b41081-1ac8-414a-9911-450a0b7b7d7f" volumeName="kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-config" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506115    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70301f22-b14c-4362-a082-15309eafa41b" volumeName="kubernetes.io/secret/70301f22-b14c-4362-a082-15309eafa41b-agent-certs" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506128    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c1e4ef-71b9-4a62-b841-a21453d83e59" volumeName="kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506141    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca506817-08d9-420e-b25d-45dc0de360c4" volumeName="kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506152    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506164    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f1ad48-874e-4463-b936-0704c67be0fd" volumeName="kubernetes.io/projected/19f1ad48-874e-4463-b936-0704c67be0fd-kube-api-access-v6ttd" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506175    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21c18bda-36de-4e16-9ea6-952ace591a1f" volumeName="kubernetes.io/secret/21c18bda-36de-4e16-9ea6-952ace591a1f-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506188    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="582ae8d7-d1b6-4066-a863-a14a5f2eb45f" volumeName="kubernetes.io/empty-dir/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-tmp-dir" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506199    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68685f74-14d9-444a-bbf1-9292f78d6a8e" volumeName="kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506212    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa65ead1-60e6-42c4-bc86-ef6df717b751" volumeName="kubernetes.io/configmap/aa65ead1-60e6-42c4-bc86-ef6df717b751-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506224    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0be5eb-c95f-462e-ab68-3e4dffaae950" volumeName="kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-tls" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506236    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506270    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea" volumeName="kubernetes.io/projected/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-kube-api-access-snsd7" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506290    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506301    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c6359e8-e292-457f-b680-f6b44bee7e79" volumeName="kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-daemon-config" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506314    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa65ead1-60e6-42c4-bc86-ef6df717b751" volumeName="kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-tls" seLinuxMountContext=""
Nov 02 13:43:43.506345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506341    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca506817-08d9-420e-b25d-45dc0de360c4" volumeName="kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506370    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b376c48-407e-4913-8b07-a0eb3fcc67a0" volumeName="kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-tmp" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506385    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506397    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f1ad48-874e-4463-b936-0704c67be0fd" volumeName="kubernetes.io/empty-dir/19f1ad48-874e-4463-b936-0704c67be0fd-audit-log" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506408    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21c18bda-36de-4e16-9ea6-952ace591a1f" volumeName="kubernetes.io/empty-dir/21c18bda-36de-4e16-9ea6-952ace591a1f-data-volume" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506421    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506433    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b652b34-c239-4b27-ada2-ec7b39d17539" volumeName="kubernetes.io/projected/7b652b34-c239-4b27-ada2-ec7b39d17539-kube-api-access-9nltd" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506444    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca506817-08d9-420e-b25d-45dc0de360c4" volumeName="kubernetes.io/projected/ca506817-08d9-420e-b25d-45dc0de360c4-kube-api-access-27t78" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506456    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506469    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c6359e8-e292-457f-b680-f6b44bee7e79" volumeName="kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506480    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b41081-1ac8-414a-9911-450a0b7b7d7f" volumeName="kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-env-overrides" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506493    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="582ae8d7-d1b6-4066-a863-a14a5f2eb45f" volumeName="kubernetes.io/secret/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-metrics-tls" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506504    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506516    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506527    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68685f74-14d9-444a-bbf1-9292f78d6a8e" volumeName="kubernetes.io/projected/68685f74-14d9-444a-bbf1-9292f78d6a8e-kube-api-access-24hl9" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506572    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506585    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0be5eb-c95f-462e-ab68-3e4dffaae950" volumeName="kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506597    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b672addc-8b6e-4983-bc93-17ff035c88bf" volumeName="kubernetes.io/configmap/b672addc-8b6e-4983-bc93-17ff035c88bf-iptables-alerter-script" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506609    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506621    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73189526-2e64-4de5-a2a9-06597d293b6b" volumeName="kubernetes.io/empty-dir/73189526-2e64-4de5-a2a9-06597d293b6b-tmp-dir" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506632    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21c18bda-36de-4e16-9ea6-952ace591a1f" volumeName="kubernetes.io/configmap/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506643    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/projected/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-kube-api-access-s2ms8" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506654    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c6359e8-e292-457f-b680-f6b44bee7e79" volumeName="kubernetes.io/projected/9c6359e8-e292-457f-b680-f6b44bee7e79-kube-api-access-rgzxj" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506667    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0be5eb-c95f-462e-ab68-3e4dffaae950" volumeName="kubernetes.io/projected/ae0be5eb-c95f-462e-ab68-3e4dffaae950-kube-api-access-tmjdj" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506680    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="582ae8d7-d1b6-4066-a863-a14a5f2eb45f" volumeName="kubernetes.io/projected/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-kube-api-access-x2b48" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506700    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b376c48-407e-4913-8b07-a0eb3fcc67a0" volumeName="kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-tuned" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506714    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa65ead1-60e6-42c4-bc86-ef6df717b751" volumeName="kubernetes.io/projected/aa65ead1-60e6-42c4-bc86-ef6df717b751-kube-api-access-9k9fw" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506726    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0be5eb-c95f-462e-ab68-3e4dffaae950" volumeName="kubernetes.io/empty-dir/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-textfile" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506738    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf3f6767-bfe3-4c84-b337-e17679488a51" volumeName="kubernetes.io/secret/cf3f6767-bfe3-4c84-b337-e17679488a51-cert" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506750    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506762    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506774    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73189526-2e64-4de5-a2a9-06597d293b6b" volumeName="kubernetes.io/projected/73189526-2e64-4de5-a2a9-06597d293b6b-kube-api-access-hnxqj" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506787    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-grpc-tls" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506799    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cccc0e-d0fc-4b34-badd-f4efbf8fd684" volumeName="kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506812    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b672addc-8b6e-4983-bc93-17ff035c88bf" volumeName="kubernetes.io/projected/b672addc-8b6e-4983-bc93-17ff035c88bf-kube-api-access-kkvvp" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506824    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca506817-08d9-420e-b25d-45dc0de360c4" volumeName="kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506835    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506847    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f1ad48-874e-4463-b936-0704c67be0fd" volumeName="kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-client-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506858    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b41081-1ac8-414a-9911-450a0b7b7d7f" volumeName="kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506869    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea" volumeName="kubernetes.io/configmap/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-serviceca" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506881    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506893    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506904    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f1ad48-874e-4463-b936-0704c67be0fd" volumeName="kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506916    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b41081-1ac8-414a-9911-450a0b7b7d7f" volumeName="kubernetes.io/secret/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506928    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="582ae8d7-d1b6-4066-a863-a14a5f2eb45f" volumeName="kubernetes.io/configmap/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-config-volume" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506940    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" volumeName="kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506951    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70301f22-b14c-4362-a082-15309eafa41b" volumeName="kubernetes.io/configmap/70301f22-b14c-4362-a082-15309eafa41b-konnectivity-ca" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506963    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0be5eb-c95f-462e-ab68-3e4dffaae950" volumeName="kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506975    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b376c48-407e-4913-8b07-a0eb3fcc67a0" volumeName="kubernetes.io/projected/8b376c48-407e-4913-8b07-a0eb3fcc67a0-kube-api-access-5wjzw" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506988    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" volumeName="kubernetes.io/configmap/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.506999    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33b41081-1ac8-414a-9911-450a0b7b7d7f" volumeName="kubernetes.io/projected/33b41081-1ac8-414a-9911-450a0b7b7d7f-kube-api-access-hnj9d" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.507011    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.507023    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9eb4eb-569f-49eb-ac33-e670af0472a9" volumeName="kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.507036    5572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f1ad48-874e-4463-b936-0704c67be0fd" volumeName="kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.507048    5572 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:43:43.507417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.507056    5572 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:43:43.517629 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.517602    5572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:43.521093 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.521063    5572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:43.527805 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.527784    5572 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 13:43:43.528340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.528322    5572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:43.533099 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.533040    5572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:43:43.534883 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.534864    5572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:43:43.534977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.534888    5572 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:43:43.534977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.534910    5572 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:43:43.534977 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:43.534949    5572 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 13:43:43.539357 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.539184    5572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:43.545940 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.545894    5572 generic.go:356] "Generic (PLEG): container finished" podID="ae0be5eb-c95f-462e-ab68-3e4dffaae950" containerID="5f2613a21d71943d2b17734bef0ab7482b2980c75923f6492f1bf77461940b2e" exitCode=0
Nov 02 13:43:43.554821 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.554791    5572 generic.go:356] "Generic (PLEG): container finished" podID="848b57801b929e97e838cbabe1dcb19f" containerID="6eb9f9faab78374ef6ea61a2e7a0b4981fcca1d0b49a5577f2407c204384f3f2" exitCode=0
Nov 02 13:43:43.578504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.578465    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a" exitCode=0
Nov 02 13:43:43.587515 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.587481    5572 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="55ecae147b737da07d455827c81533633c262b87d784294648d8c3c5e468e285" exitCode=0
Nov 02 13:43:43.587515 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.587510    5572 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="0cdfae6f5a9361612d6d05128b0aee33b5d46088d7f38478705e2e5000d92604" exitCode=0
Nov 02 13:43:43.587672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.587519    5572 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="1f45c016b284939bd18d3d1a8b5a0e606845858ef0650da6e096ec0675ea0285" exitCode=0
Nov 02 13:43:43.587672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.587528    5572 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="2ceaaba45b723a9bf6dc833075d09c310ae2b1edc77ccc6a345e377129bca1b0" exitCode=0
Nov 02 13:43:43.587672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.587538    5572 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="96d37d63616744e46922f2d8e36fcdcadbff1f37248f025957717d805eff4422" exitCode=0
Nov 02 13:43:43.587672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.587549    5572 generic.go:356] "Generic (PLEG): container finished" podID="ca506817-08d9-420e-b25d-45dc0de360c4" containerID="e15a2d3efac5d82736b3d8f28f32e40d52c8382dfb298059d559dcc51bb3320f" exitCode=0
Nov 02 13:43:43.608368 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.608325    5572 manager.go:324] Recovery completed
Nov 02 13:43:43.629130 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629105    5572 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:43:43.629130 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629128    5572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:43:43.629257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629147    5572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:43:43.629331 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629312    5572 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 13:43:43.629377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629333    5572 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 13:43:43.629439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629429    5572 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 13:43:43.629488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629441    5572 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 13:43:43.629488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629449    5572 policy_none.go:50] "Start"
Nov 02 13:43:43.629488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629473    5572 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:43:43.629488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629486    5572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:43:43.629732 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629718    5572 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 13:43:43.629782 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629734    5572 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 13:43:43.629782 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.629743    5572 policy_none.go:44] "Start"
Nov 02 13:43:43.634461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.634436    5572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:43:43.634997 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:43.634979    5572 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 13:43:43.637003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.636991    5572 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:43:43.637486 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:43.637014    5572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:43:43.637550 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.637491    5572 server.go:87] "Starting device plugin registration server"
Nov 02 13:43:43.637768 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.637757    5572 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:43:43.637811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.637769    5572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:43:43.637872 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.637860    5572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:43:43.637970 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.637942    5572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:43:43.638076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.637978    5572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:43:43.638076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.637984    5572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:43:43.638181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.638024    5572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:43:43.637964956Z","UUID":"f04e4769-8672-4419-9027-57da46000e20","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:43:43.639646 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:43.639627    5572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:43:43.641710 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.641681    5572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:43:43.641802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.641731    5572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:43:43.738848 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.738823    5572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:43.740614 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.740597    5572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:43.740682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.740624    5572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:43.740682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.740634    5572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-62.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 13:43:43.740682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.740668    5572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.769245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.769225    5572 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.769316 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.769276    5572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.835879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.835841    5572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"]
Nov 02 13:43:43.836431 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.836418    5572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.836467 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.836452    5572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.836531 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.836494    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"848b57801b929e97e838cbabe1dcb19f","Type":"ContainerStarted","Data":"b27e5a2024ae1714427b6838f1fe701e9f2af8dc117b4bcc3fe1c060a83222dd"}
Nov 02 13:43:43.836592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.836551    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"848b57801b929e97e838cbabe1dcb19f","Type":"ContainerDied","Data":"6eb9f9faab78374ef6ea61a2e7a0b4981fcca1d0b49a5577f2407c204384f3f2"}
Nov 02 13:43:43.836634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.836600    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"848b57801b929e97e838cbabe1dcb19f","Type":"ContainerStarted","Data":"87115514af2f599dede11b71672dff89109909da9b998e0ffdb71c478a21ac30"}
Nov 02 13:43:43.836634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.836618    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"b4ea240f21eef62637ef08b4bbcf8f6a","Type":"ContainerStarted","Data":"b5a9dae04a5a9557ed3fb40da55b1c43ac2f7b90112dfe03332b04b20819917a"}
Nov 02 13:43:43.836634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.836632    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal" event={"ID":"b4ea240f21eef62637ef08b4bbcf8f6a","Type":"ContainerStarted","Data":"eb166106f9470b20af2a663332e867b4db3f2574e19a6600435c70b8d131cecc"}
Nov 02 13:43:43.849696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.849677    5572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:43.849733 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:43.849713    5572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.850464 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.850451    5572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:43.850496 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:43.850484    5572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.930843 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.930787    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4ea240f21eef62637ef08b4bbcf8f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"b4ea240f21eef62637ef08b4bbcf8f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.930843 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.930815    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:43.930843 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:43.930831    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.031379 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.031346    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.031379 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.031377    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4ea240f21eef62637ef08b4bbcf8f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"b4ea240f21eef62637ef08b4bbcf8f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.031523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.031396    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.031523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.031435    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.031523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.031450    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/848b57801b929e97e838cbabe1dcb19f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"848b57801b929e97e838cbabe1dcb19f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.031523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.031456    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4ea240f21eef62637ef08b4bbcf8f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" (UID: \"b4ea240f21eef62637ef08b4bbcf8f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.486046 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.486015    5572 apiserver.go:52] "Watching apiserver"
Nov 02 13:43:44.497734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.497711    5572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:43:44.498284 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.498262    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dd7f9986f-b8g5g","openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr","kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal","openshift-monitoring/thanos-querier-55c454df6-wgrrz","openshift-network-operator/iptables-alerter-g2xz6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht","openshift-monitoring/alertmanager-main-0","openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c","kube-system/global-pull-secret-syncer-q6z7m","openshift-cluster-node-tuning-operator/tuned-wsn55","openshift-dns/dns-default-29msq","openshift-insights/insights-runtime-extractor-8gf9j","openshift-monitoring/node-exporter-tczrv","openshift-network-diagnostics/network-check-target-gbsrr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal","openshift-multus/multus-grrwm","openshift-ovn-kubernetes/ovnkube-node-2qbkw","openshift-ingress-canary/ingress-canary-vpzfd","kube-system/konnectivity-agent-jt476","openshift-dns/node-resolver-sfqfd","openshift-multus/multus-additional-cni-plugins-jrphq","openshift-multus/network-metrics-daemon-l7vld","openshift-image-registry/node-ca-cxttx"]
Nov 02 13:43:44.503976 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.503958    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.504181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.504168    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.504730 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.504712    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:43:44.507710 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.507685    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:43:44.507890 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.507865    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzjbd\""
Nov 02 13:43:44.508105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.508080    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n56j5\""
Nov 02 13:43:44.508404 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.508383    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s2t6n\""
Nov 02 13:43:44.508497 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.508481    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:43:44.509157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.509133    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.509503 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.509484    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.509687 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.509664    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.509774 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.509755    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.509969 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.509955    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.510188 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510173    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.510367 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510346    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.510421 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510407    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:43:44.510604 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510582    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.510734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510705    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.510816 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510707    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:43:44.510867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510734    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:43:44.510913 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.510765    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:43:44.511652 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511630    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:43:44.511725 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511689    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.511782 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511726    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.511782 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511776    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m5gp5\""
Nov 02 13:43:44.511871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511821    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tb7ws\""
Nov 02 13:43:44.511871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511861    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8rzz2\""
Nov 02 13:43:44.511944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511885    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.511982 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511954    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:43:44.512031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.511996    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:43:44.512031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512023    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 13:43:44.512125 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512028    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qngrq\""
Nov 02 13:43:44.512125 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512065    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjbpx\""
Nov 02 13:43:44.512288 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512266    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:43:44.512392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512320    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:43:44.512392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512333    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cw7nr\""
Nov 02 13:43:44.512392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512364    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:43:44.512548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512390    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 13:43:44.512548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512420    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:43:44.512548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512422    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:43:44.512548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512449    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.512548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512473    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mlv7f\""
Nov 02 13:43:44.512548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512522    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:43:44.512548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512543    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cfdrs\""
Nov 02 13:43:44.512889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512598    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.512889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512664    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.512889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512706    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.512889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512775    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9j4gh\""
Nov 02 13:43:44.512889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512836    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:43:44.512889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512847    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.513152 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512894    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.513152 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512929    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 13:43:44.513152 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.512961    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cmphd\""
Nov 02 13:43:44.513152 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513000    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:43:44.513152 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513041    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fbcvm\""
Nov 02 13:43:44.513152 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513049    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:43:44.513152 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513126    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:43:44.513479 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513160    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sw4zl\""
Nov 02 13:43:44.513479 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513216    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-clq6z\""
Nov 02 13:43:44.513479 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513248    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:43:44.513650 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.513629    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:43:44.532949 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.532925    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:44.533058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.532956    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-ovn\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.533058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.532975    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-node-log\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.533058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.532989    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73189526-2e64-4de5-a2a9-06597d293b6b-hosts-file\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:43:44.533058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533017    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-kubelet-config\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.533276 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533078    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysconfig\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.533276 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533124    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-cnibin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.533276 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533163    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68685f74-14d9-444a-bbf1-9292f78d6a8e-metrics-certs\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:44.533276 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533183    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-etc-kubernetes\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.533276 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533209    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-sys\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.533276 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533240    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hl9\" (UniqueName: \"kubernetes.io/projected/68685f74-14d9-444a-bbf1-9292f78d6a8e-kube-api-access-24hl9\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533303    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-config\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533387    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73189526-2e64-4de5-a2a9-06597d293b6b-tmp-dir\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533426    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-netns\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533450    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzxj\" (UniqueName: \"kubernetes.io/projected/9c6359e8-e292-457f-b680-f6b44bee7e79-kube-api-access-rgzxj\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533492    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73189526-2e64-4de5-a2a9-06597d293b6b-tmp-dir\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533491    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-config\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533501    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533531    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-multus-certs\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.533593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533571    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnj9d\" (UniqueName: \"kubernetes.io/projected/33b41081-1ac8-414a-9911-450a0b7b7d7f-kube-api-access-hnj9d\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533603    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-daemon-config\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533660    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-kubernetes\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533688    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-run\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533724    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-systemd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533758    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-script-lib\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533776    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-daemon-config\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533829    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-serviceca\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533865    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-cni-binary-copy\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533924    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-system-cni-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533933    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovnkube-script-lib\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533959    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.533986    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-socket-dir-parent\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534018    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-kubelet\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534043    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-kubelet\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534060    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c6359e8-e292-457f-b680-f6b44bee7e79-cni-binary-copy\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534066    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-systemd-units\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534112    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-env-overrides\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534145    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovn-node-metrics-cert\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534185    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsd7\" (UniqueName: \"kubernetes.io/projected/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-kube-api-access-snsd7\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534231    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-socket-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534249    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33b41081-1ac8-414a-9911-450a0b7b7d7f-env-overrides\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534256    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534329    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-tuned\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534341    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33b41081-1ac8-414a-9911-450a0b7b7d7f-ovn-node-metrics-cert\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534401    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-dbus\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534420    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-system-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534433    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-serviceca\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534445    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-conf-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534471    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-tuned\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534518    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-device-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534549    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70301f22-b14c-4362-a082-15309eafa41b-konnectivity-ca\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534600    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534618    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-log-socket\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534641    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534667    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534693    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nltd\" (UniqueName: \"kubernetes.io/projected/7b652b34-c239-4b27-ada2-ec7b39d17539-kube-api-access-9nltd\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534706    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70301f22-b14c-4362-a082-15309eafa41b-konnectivity-ca\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534718    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-modprobe-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534742    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-conf\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534764    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-cnibin\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534794    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534870    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534898    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-var-lib-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.534926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.534918    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535133    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535174    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-multus\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535201    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-registration-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535224    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b672addc-8b6e-4983-bc93-17ff035c88bf-host-slash\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535266    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-tmp\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535361    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b376c48-407e-4913-8b07-a0eb3fcc67a0-tmp\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535396    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535417    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-sys-fs\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535435    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvvp\" (UniqueName: \"kubernetes.io/projected/b672addc-8b6e-4983-bc93-17ff035c88bf-kube-api-access-kkvvp\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535450    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-lib-modules\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535464    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-var-lib-kubelet\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535492    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535514    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-bin\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535531    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-netd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535637    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535677    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-host\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535707    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjzw\" (UniqueName: \"kubernetes.io/projected/8b376c48-407e-4913-8b07-a0eb3fcc67a0-kube-api-access-5wjzw\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535729    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca506817-08d9-420e-b25d-45dc0de360c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535734    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27t78\" (UniqueName: \"kubernetes.io/projected/ca506817-08d9-420e-b25d-45dc0de360c4-kube-api-access-27t78\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535763    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-slash\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.535870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535873    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxqj\" (UniqueName: \"kubernetes.io/projected/73189526-2e64-4de5-a2a9-06597d293b6b-kube-api-access-hnxqj\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535975    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-k8s-cni-cncf-io\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.535992    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-hostroot\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536008    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536023    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70301f22-b14c-4362-a082-15309eafa41b-agent-certs\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536039    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b672addc-8b6e-4983-bc93-17ff035c88bf-iptables-alerter-script\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536071    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-host\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536147    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-bin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536196    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-netns\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536221    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-etc-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536257    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536262    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70301f22-b14c-4362-a082-15309eafa41b-agent-certs\") pod \"konnectivity-agent-jt476\" (UID: \"70301f22-b14c-4362-a082-15309eafa41b\") " pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536283    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b672addc-8b6e-4983-bc93-17ff035c88bf-iptables-alerter-script\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536294    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-systemd\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.536530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.536512    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c1e4ef-71b9-4a62-b841-a21453d83e59-original-pull-secret\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.549399 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.549381    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjzw\" (UniqueName: \"kubernetes.io/projected/8b376c48-407e-4913-8b07-a0eb3fcc67a0-kube-api-access-5wjzw\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.549490 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.549394    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27t78\" (UniqueName: \"kubernetes.io/projected/ca506817-08d9-420e-b25d-45dc0de360c4-kube-api-access-27t78\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.549658 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.549644    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hl9\" (UniqueName: \"kubernetes.io/projected/68685f74-14d9-444a-bbf1-9292f78d6a8e-kube-api-access-24hl9\") pod \"network-metrics-daemon-l7vld\" (UID: \"68685f74-14d9-444a-bbf1-9292f78d6a8e\") " pod="openshift-multus/network-metrics-daemon-l7vld"
Nov 02 13:43:44.553154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.553132    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nltd\" (UniqueName: \"kubernetes.io/projected/7b652b34-c239-4b27-ada2-ec7b39d17539-kube-api-access-9nltd\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.553246 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.553195    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnj9d\" (UniqueName: \"kubernetes.io/projected/33b41081-1ac8-414a-9911-450a0b7b7d7f-kube-api-access-hnj9d\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.557419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.557397    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsd7\" (UniqueName: \"kubernetes.io/projected/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-kube-api-access-snsd7\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:43:44.557507 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.557420    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbql\" (UniqueName: \"kubernetes.io/projected/91cccc0e-d0fc-4b34-badd-f4efbf8fd684-kube-api-access-vjbql\") pod \"network-check-target-gbsrr\" (UID: \"91cccc0e-d0fc-4b34-badd-f4efbf8fd684\") " pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:43:44.557507 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.557455    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvvp\" (UniqueName: \"kubernetes.io/projected/b672addc-8b6e-4983-bc93-17ff035c88bf-kube-api-access-kkvvp\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:43:44.557684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.557670    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzxj\" (UniqueName: \"kubernetes.io/projected/9c6359e8-e292-457f-b680-f6b44bee7e79-kube-api-access-rgzxj\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.559068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.559049    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxqj\" (UniqueName: \"kubernetes.io/projected/73189526-2e64-4de5-a2a9-06597d293b6b-kube-api-access-hnxqj\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:43:44.593647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.593624    5572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.593810 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.593741    5572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.603811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.603790    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 13:43:44.603811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.603790    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 13:43:44.603811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.603816    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 13:43:44.604014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.603859    5572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:43:44.606247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606225    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 13:43:44.606342 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606285    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 13:43:44.606342 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606338    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 13:43:44.606445 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606385    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 13:43:44.606445 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606423    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 13:43:44.606532 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606506    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 13:43:44.606666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606626    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 13:43:44.606767 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606704    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 13:43:44.606767 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606741    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 13:43:44.606767 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606755    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 13:43:44.606767 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606764    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606796    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606825    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606825    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606842    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606865    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606866    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606883    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-nsff4rjr46lv\""
Nov 02 13:43:44.607045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.606931    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f2hpb\""
Nov 02 13:43:44.607417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.607195    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gnlzz\""
Nov 02 13:43:44.607417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.607209    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4c928\""
Nov 02 13:43:44.607417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.607248    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x2jws\""
Nov 02 13:43:44.607417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.607364    5572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:44.607417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.607364    5572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:44.607417 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.607414    5572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.607657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.607417    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 13:43:44.607657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.607489    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-epoqrs26tg7nr\""
Nov 02 13:43:44.607657 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.607618    5572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal"
Nov 02 13:43:44.636962 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.636932    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-tmp\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.636962 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.636966    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-client-certs\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.637210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.636984    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ttd\" (UniqueName: \"kubernetes.io/projected/19f1ad48-874e-4463-b936-0704c67be0fd-kube-api-access-v6ttd\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.637210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637001    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.637210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637053    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.637210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637091    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-tmp\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.637210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637142    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-lib-modules\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.637210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637186    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637212    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637238    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637260    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-client-certs\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637264    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-host\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637298    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-lib-modules\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637314    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea-host\") pod \"node-ca-cxttx\" (UID: \"5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea\") " pod="openshift-image-registry/node-ca-cxttx"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637313    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19f1ad48-874e-4463-b936-0704c67be0fd-audit-log\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637360    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637371    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/19f1ad48-874e-4463-b936-0704c67be0fd-audit-log\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637389    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21c18bda-36de-4e16-9ea6-952ace591a1f-crio-socket\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637387    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637415    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21c18bda-36de-4e16-9ea6-952ace591a1f-data-volume\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637472    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637488    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21c18bda-36de-4e16-9ea6-952ace591a1f-data-volume\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637495    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.637544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637534    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637593    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-client-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637629    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-bin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637636    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637655    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-systemd\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637679    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-node-log\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637691    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-bin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637707    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73189526-2e64-4de5-a2a9-06597d293b6b-hosts-file\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637730    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-cnibin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637728    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637741    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-systemd\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637744    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-node-log\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637745    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-etc-kubernetes\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637768    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-etc-kubernetes\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637773    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73189526-2e64-4de5-a2a9-06597d293b6b-hosts-file\") pod \"node-resolver-sfqfd\" (UID: \"73189526-2e64-4de5-a2a9-06597d293b6b\") " pod="openshift-dns/node-resolver-sfqfd"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637792    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-cnibin\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637792    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q6p\" (UniqueName: \"kubernetes.io/projected/cf3f6767-bfe3-4c84-b337-e17679488a51-kube-api-access-97q6p\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637822    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-tmp-dir\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637827    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-client-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637838    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637854    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-textfile\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637877    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-wtmp\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637903    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-metrics-server-audit-profiles\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637908    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-tmp-dir\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.637926    5572 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637931    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-netns\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637935    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-textfile\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.637984    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjdj\" (UniqueName: \"kubernetes.io/projected/ae0be5eb-c95f-462e-ab68-3e4dffaae950-kube-api-access-tmjdj\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.637997    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:45.137978309 +0000 UTC m=+1.716067172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"service-ca" not registered
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638065    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-netns\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638076    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638108    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638137    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-run\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638164    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638193    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-tls\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638231    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-run\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638239    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.638250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638245    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638289    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-metrics-server-audit-profiles\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638328    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638367    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638373    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638386    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-socket-dir-parent\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638402    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-kubelet\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638418    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b48\" (UniqueName: \"kubernetes.io/projected/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-kube-api-access-x2b48\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638439    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-socket-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638455    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638459    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-tls\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638470    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-kubelet\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638479    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638471    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-socket-dir-parent\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638463    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-device-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638505    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-socket-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638517    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-device-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638518    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-grpc-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638587    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638623    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrfg\" (UniqueName: \"kubernetes.io/projected/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-api-access-6nrfg\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638647    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-cnibin\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638671    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-registration-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638696    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-cnibin\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638698    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638724    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638722    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa65ead1-60e6-42c4-bc86-ef6df717b751-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638731    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-registration-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638770    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638785    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-grpc-tls\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638798    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21c18bda-36de-4e16-9ea6-952ace591a1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638838    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-sys-fs\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638864    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-var-lib-kubelet\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638897    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-accelerators-collector-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638924    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-sys-fs\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638931    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638955    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638959    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-bin\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638977    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa65ead1-60e6-42c4-bc86-ef6df717b751-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638976    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-bin\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.638928    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-var-lib-kubelet\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639003    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-netd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639003    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639028    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21c18bda-36de-4e16-9ea6-952ace591a1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639035    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-cni-netd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639029    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639072    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639075    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639091    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-accelerators-collector-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639183    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-tls\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639212    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-slash\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639229    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-k8s-cni-cncf-io\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639244    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-hostroot\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639268    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-metrics-client-ca\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639287    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-slash\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639292    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-host\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639298    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-k8s-cni-cncf-io\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639308    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-hostroot\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639335    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9fw\" (UniqueName: \"kubernetes.io/projected/aa65ead1-60e6-42c4-bc86-ef6df717b751-kube-api-access-9k9fw\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639342    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/19f1ad48-874e-4463-b936-0704c67be0fd-secret-metrics-server-tls\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639338    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-host\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639364    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs222\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639383    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-netns\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639403    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-etc-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639422    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf3f6767-bfe3-4c84-b337-e17679488a51-cert\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639421    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-netns\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639445    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-metrics-client-ca\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639453    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-etc-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639382    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639474    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-ovn\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639495    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-kubelet-config\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639511    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysconfig\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639525    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639542    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-ovn\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639588    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-kubelet-config\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639600    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysconfig\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639612    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.639618    5572 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639637    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-sys\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639644    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf3f6767-bfe3-4c84-b337-e17679488a51-cert\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639655    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639672    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-metrics-client-ca\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639687    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639688    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-sys\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.639704    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:45.139685414 +0000 UTC m=+1.717774261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.639704    5572 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639731    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.639747    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:45.139732155 +0000 UTC m=+1.717821016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-config" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639760    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639775    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639804    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-multus-certs\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639818    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639833    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639838    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ms8\" (UniqueName: \"kubernetes.io/projected/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-kube-api-access-s2ms8\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639863    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639869    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-run-multus-certs\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639875    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0be5eb-c95f-462e-ab68-3e4dffaae950-metrics-client-ca\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639885    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-kubernetes\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639901    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-config-volume\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639916    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-systemd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639933    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639940    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-kubernetes\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639955    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639986    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-run-systemd\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.639990    5572 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640017    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640030    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f1ad48-874e-4463-b936-0704c67be0fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.640037    5572 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.639995    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640070    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-config-volume\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.640041    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:45.140027121 +0000 UTC m=+1.718115985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.640125    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:45.140110232 +0000 UTC m=+1.718199082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640144    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640176    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-system-cni-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640199    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa65ead1-60e6-42c4-bc86-ef6df717b751-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.640132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640201    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-metrics-tls\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640231    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640243    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca506817-08d9-420e-b25d-45dc0de360c4-system-cni-dir\") pod \"multus-additional-cni-plugins-jrphq\" (UID: \"ca506817-08d9-420e-b25d-45dc0de360c4\") " pod="openshift-multus/multus-additional-cni-plugins-jrphq"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640256    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640282    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640308    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.640312    5572 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640312    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-sys\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640362    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-kubelet\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640368    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640377    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-metrics-tls\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640391    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-systemd-units\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640405    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-kubelet\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640425    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-systemd-units\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640434    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.640476    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:45.140463546 +0000 UTC m=+1.718552406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640478    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640509    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640608    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640609    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-dbus\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640643    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-system-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640646    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640666    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-conf-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640687    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-root\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640709    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-log-socket\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640712    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-system-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640720    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-conf-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640744    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c1e4ef-71b9-4a62-b841-a21453d83e59-dbus\") pod \"global-pull-secret-syncer-q6z7m\" (UID: \"71c1e4ef-71b9-4a62-b841-a21453d83e59\") " pod="kube-system/global-pull-secret-syncer-q6z7m"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640751    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640761    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-log-socket\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640780    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640794    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-multus-cni-dir\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640800    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-modprobe-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640815    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-conf\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640832    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-var-lib-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640855    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640875    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-multus\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640879    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-var-lib-openvswitch\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640900    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b652b34-c239-4b27-ada2-ec7b39d17539-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vcht\" (UID: \"7b652b34-c239-4b27-ada2-ec7b39d17539\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vcht"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640906    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b672addc-8b6e-4983-bc93-17ff035c88bf-host-slash\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640914    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c6359e8-e292-457f-b680-f6b44bee7e79-host-var-lib-cni-multus\") pod \"multus-grrwm\" (UID: \"9c6359e8-e292-457f-b680-f6b44bee7e79\") " pod="openshift-multus/multus-grrwm"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640908    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-modprobe-d\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640933    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640941    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b672addc-8b6e-4983-bc93-17ff035c88bf-host-slash\") pod \"iptables-alerter-g2xz6\" (UID: \"b672addc-8b6e-4983-bc93-17ff035c88bf\") " pod="openshift-network-operator/iptables-alerter-g2xz6"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640967    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b376c48-407e-4913-8b07-a0eb3fcc67a0-etc-sysctl-conf\") pod \"tuned-wsn55\" (UID: \"8b376c48-407e-4913-8b07-a0eb3fcc67a0\") " pod="openshift-cluster-node-tuning-operator/tuned-wsn55"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.640937    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33b41081-1ac8-414a-9911-450a0b7b7d7f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qbkw\" (UID: \"33b41081-1ac8-414a-9911-450a0b7b7d7f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:44.643321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.641089    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.652377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.652361    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrfg\" (UniqueName: \"kubernetes.io/projected/21c18bda-36de-4e16-9ea6-952ace591a1f-kube-api-access-6nrfg\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.652453 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.652430    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ttd\" (UniqueName: \"kubernetes.io/projected/19f1ad48-874e-4463-b936-0704c67be0fd-kube-api-access-v6ttd\") pod \"metrics-server-7b5cc8db4b-7gz9c\" (UID: \"19f1ad48-874e-4463-b936-0704c67be0fd\") " pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:43:44.653278 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.653257    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9fw\" (UniqueName: \"kubernetes.io/projected/aa65ead1-60e6-42c4-bc86-ef6df717b751-kube-api-access-9k9fw\") pod \"prometheus-operator-9cd89b74c-zpxdr\" (UID: \"aa65ead1-60e6-42c4-bc86-ef6df717b751\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zpxdr"
Nov 02 13:43:44.656263 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.656247    5572 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 13:43:44.656358 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.656269    5572 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 13:43:44.656358 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.656280    5572 projected.go:196] Error preparing data for projected volume kube-api-access-w2hgh for pod openshift-console/console-7dd7f9986f-b8g5g: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:44.656358 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:44.656316    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:45.156306669 +0000 UTC m=+1.734395513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w2hgh" (UniqueName: "kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:44.657321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.657298    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ms8\" (UniqueName: \"kubernetes.io/projected/8cfd9f5a-4325-4769-9b66-8ef4dc715b5c-kube-api-access-s2ms8\") pod \"thanos-querier-55c454df6-wgrrz\" (UID: \"8cfd9f5a-4325-4769-9b66-8ef4dc715b5c\") " pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:44.657441 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.657420    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs222\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\") pod \"alertmanager-main-0\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:44.657628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.657613    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2b48\" (UniqueName: \"kubernetes.io/projected/582ae8d7-d1b6-4066-a863-a14a5f2eb45f-kube-api-access-x2b48\") pod \"dns-default-29msq\" (UID: \"582ae8d7-d1b6-4066-a863-a14a5f2eb45f\") " pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:44.658272 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.658257    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q6p\" (UniqueName: \"kubernetes.io/projected/cf3f6767-bfe3-4c84-b337-e17679488a51-kube-api-access-97q6p\") pod \"ingress-canary-vpzfd\" (UID: \"cf3f6767-bfe3-4c84-b337-e17679488a51\") " pod="openshift-ingress-canary/ingress-canary-vpzfd"
Nov 02 13:43:44.662278 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.662263    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjdj\" (UniqueName: \"kubernetes.io/projected/ae0be5eb-c95f-462e-ab68-3e4dffaae950-kube-api-access-tmjdj\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.741893 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.741831    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-wtmp\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.742025 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.741934    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-sys\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.742025 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.741954    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-root\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.742025 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.741953    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-node-exporter-wtmp\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.742025 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.741980    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21c18bda-36de-4e16-9ea6-952ace591a1f-crio-socket\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:44.742025 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.741984    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-sys\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.742025 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.741996    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0be5eb-c95f-462e-ab68-3e4dffaae950-root\") pod \"node-exporter-tczrv\" (UID: \"ae0be5eb-c95f-462e-ab68-3e4dffaae950\") " pod="openshift-monitoring/node-exporter-tczrv"
Nov 02 13:43:44.742205 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:44.742033    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21c18bda-36de-4e16-9ea6-952ace591a1f-crio-socket\") pod \"insights-runtime-extractor-8gf9j\" (UID: \"21c18bda-36de-4e16-9ea6-952ace591a1f\") " pod="openshift-insights/insights-runtime-extractor-8gf9j"
Nov 02 13:43:45.039650 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.039573    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz"
Nov 02 13:43:45.146073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.146042    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:45.146073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.146079    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.146095    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.146114    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146136    5572 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.146172    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146176    5572 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146215    5572 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146226    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:46.146207782 +0000 UTC m=+2.724296647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-config" not registered
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146193    5572 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146254    5572 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146260    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:46.146246076 +0000 UTC m=+2.724334927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:45.146317 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146310    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:46.146293937 +0000 UTC m=+2.724382795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:45.146740 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146332    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:46.146320485 +0000 UTC m=+2.724409331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:45.146740 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146351    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:46.146342427 +0000 UTC m=+2.724431277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"service-ca" not registered
Nov 02 13:43:45.146740 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.146390    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:45.146740 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146456    5572 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:45.146740 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.146484    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:46.146477203 +0000 UTC m=+2.724566048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:45.247447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.247412    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:45.247642 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.247617    5572 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 13:43:45.247642 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.247642    5572 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 13:43:45.247768 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.247655    5572 projected.go:196] Error preparing data for projected volume kube-api-access-w2hgh for pod openshift-console/console-7dd7f9986f-b8g5g: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:45.247768 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:45.247726    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:46.247706844 +0000 UTC m=+2.825795693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2hgh" (UniqueName: "kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:45.256378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:45.256358    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-29msq"
Nov 02 13:43:46.156728 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:46.156675    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:46.156763    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156806    5572 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:46.156836    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:46.156863    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156870    5572 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156883    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:48.156868329 +0000 UTC m=+4.734957177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156918    5572 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:46.156928    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156933    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:48.156921176 +0000 UTC m=+4.735010060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"service-ca" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:46.156959    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156977    5572 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156990    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:48.156975459 +0000 UTC m=+4.735064326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-config" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.156993    5572 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.157027    5572 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.157052    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:48.157032727 +0000 UTC m=+4.735121584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.157067    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:48.157060929 +0000 UTC m=+4.735149773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:46.157174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.157076    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:48.157071106 +0000 UTC m=+4.735159951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:46.257894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:46.257870    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:46.257997 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.257972    5572 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 13:43:46.257997 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.257983    5572 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 13:43:46.257997 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.257992    5572 projected.go:196] Error preparing data for projected volume kube-api-access-w2hgh for pod openshift-console/console-7dd7f9986f-b8g5g: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:46.258097 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:46.258028    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:48.258019373 +0000 UTC m=+4.836108218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2hgh" (UniqueName: "kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:48.177014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:48.176976    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:48.177042    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177122    5572 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177119    5572 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177170    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:52.177158125 +0000 UTC m=+8.755246970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177199    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:52.177178214 +0000 UTC m=+8.755267063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"service-ca" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:48.177232    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:48.177283    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:48.177309    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177332    5572 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:48.177342    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177399    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:52.177381824 +0000 UTC m=+8.755470683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-config" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177416    5572 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177435    5572 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177434    5572 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177460    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:52.177447065 +0000 UTC m=+8.755535916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177482    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:52.177470589 +0000 UTC m=+8.755559485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:48.177534 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.177497    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:52.177489945 +0000 UTC m=+8.755578796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:48.278282 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:48.278252    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:48.278420 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.278386    5572 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 13:43:48.278420 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.278403    5572 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 13:43:48.278420 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.278415    5572 projected.go:196] Error preparing data for projected volume kube-api-access-w2hgh for pod openshift-console/console-7dd7f9986f-b8g5g: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:48.278618 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:48.278470    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:52.278454902 +0000 UTC m=+8.856543766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2hgh" (UniqueName: "kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:51.428439 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:51.428399    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:43:51.833610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:51.833583    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:51.873011 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:51.872985    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jt476"
Nov 02 13:43:52.215180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.215087    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:52.215180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.215128    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:52.215180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.215150    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215205    5572 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215237    5572 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.215276    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215305    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:00.215291836 +0000 UTC m=+16.793380703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215325    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:00.215319575 +0000 UTC m=+16.793408422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-config" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.215350    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215364    5572 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215370    5572 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.215399    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215417    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:00.215410783 +0000 UTC m=+16.793499628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215436    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:00.215424433 +0000 UTC m=+16.793513290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215450    5572 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 13:43:52.215469 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215454    5572 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:52.215893 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215492    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:00.21547691 +0000 UTC m=+16.793565755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"service-ca" not registered
Nov 02 13:43:52.215893 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.215510    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:00.215498954 +0000 UTC m=+16.793587816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:43:52.316504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.316476    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:43:52.316629 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.316614    5572 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 13:43:52.316680 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.316630    5572 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 13:43:52.316680 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.316639    5572 projected.go:196] Error preparing data for projected volume kube-api-access-w2hgh for pod openshift-console/console-7dd7f9986f-b8g5g: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:52.316736 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:43:52.316686    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:00.316673945 +0000 UTC m=+16.894762793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2hgh" (UniqueName: "kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:52.771508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.771484    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:43:52.771880 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.771687    5572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:52.877236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:43:52.877199    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:44:00.289971 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:00.289932    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:00.290021    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:00.290047    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:00.290080    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290047    5572 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:00.290107    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:00.290140    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290109    5572 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290177    5572 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290186    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:16.290165827 +0000 UTC m=+32.868254688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"service-ca" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290196    5572 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290135    5572 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290225    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:16.290213801 +0000 UTC m=+32.868302646 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290141    5572 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290241    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:16.290232988 +0000 UTC m=+32.868321832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-serving-cert" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290256    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:16.290247504 +0000 UTC m=+32.868336351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-config" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290270    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:16.290262636 +0000 UTC m=+32.868351486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"console-oauth-config" not registered
Nov 02 13:44:00.290524 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.290284    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:16.29027612 +0000 UTC m=+32.868364994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 02 13:44:00.391258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:00.391225    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"console-7dd7f9986f-b8g5g\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") " pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:00.391423 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.391404    5572 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 02 13:44:00.391468 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.391428    5572 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 02 13:44:00.391468 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.391440    5572 projected.go:196] Error preparing data for projected volume kube-api-access-w2hgh for pod openshift-console/console-7dd7f9986f-b8g5g: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:44:00.391531 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:00.391503    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh podName:62b7284f-c8bf-4bd4-8b71-16b297e9d2e8 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:16.391489363 +0000 UTC m=+32.969578207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2hgh" (UniqueName: "kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh") pod "console-7dd7f9986f-b8g5g" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 02 13:44:05.428287 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:05.428259    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b5cc8db4b-7gz9c"
Nov 02 13:44:09.530074 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.530026    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7dd7f9986f-b8g5g" podUID="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" containerName="console" containerID="cri-o://8b3992990066f02c0a93407197f11fcf853eb899ba3ec186dc4d108812d5b4fd" gracePeriod=15
Nov 02 13:44:09.676930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.676897    5572 generic.go:356] "Generic (PLEG): container finished" podID="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" containerID="8b3992990066f02c0a93407197f11fcf853eb899ba3ec186dc4d108812d5b4fd" exitCode=0
Nov 02 13:44:09.677097 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.676972    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7f9986f-b8g5g" event={"ID":"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8","Type":"ContainerDied","Data":"8b3992990066f02c0a93407197f11fcf853eb899ba3ec186dc4d108812d5b4fd"}
Nov 02 13:44:09.785109 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.785059    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:09.877072 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877042    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877113    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877175    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877199    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877222    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877240    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877257    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877274    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") pod \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\" (UID: \"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8\") "
Nov 02 13:44:09.877423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877344    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:09.877705 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877455    5572 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-tmp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:09.877705 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877599    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:09.877816 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877790    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:09.877816 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877797    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:09.877816 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.877804    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:09.879575 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.879543    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "kube-api-access-w2hgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:09.879714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.879696    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:09.879866 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.879850    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert" pod "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" (UID: "62b7284f-c8bf-4bd4-8b71-16b297e9d2e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:09.978499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.978461    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hgh\" (UniqueName: \"kubernetes.io/projected/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-kube-api-access-w2hgh\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:09.978499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.978492    5572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-oauth-config\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:09.978499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.978503    5572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-config\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:09.978757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.978513    5572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-trusted-ca-bundle\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:09.978757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.978522    5572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-oauth-serving-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:09.978757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.978530    5572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-service-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:09.978757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:09.978538    5572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8-console-serving-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:10.659535 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.659505    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:10.659981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.659808    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="alertmanager" containerID="cri-o://ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0" gracePeriod=120
Nov 02 13:44:10.659981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.659886    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-metric" containerID="cri-o://324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7" gracePeriod=120
Nov 02 13:44:10.659981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.659917    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-web" containerID="cri-o://98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c" gracePeriod=120
Nov 02 13:44:10.659981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.659936    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="prom-label-proxy" containerID="cri-o://1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe" gracePeriod=120
Nov 02 13:44:10.660199 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.659963    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="config-reloader" containerID="cri-o://012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5" gracePeriod=120
Nov 02 13:44:10.660199 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.659946    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy" containerID="cri-o://41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39" gracePeriod=120
Nov 02 13:44:10.681950 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.681923    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7f9986f-b8g5g" event={"ID":"62b7284f-c8bf-4bd4-8b71-16b297e9d2e8","Type":"ContainerDied","Data":"4898619cf29b85246716584c663cbb8a963b2dd54873fc7e5481399b7b658383"}
Nov 02 13:44:10.682048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.681990    5572 scope.go:122] "RemoveContainer" containerID="8b3992990066f02c0a93407197f11fcf853eb899ba3ec186dc4d108812d5b4fd"
Nov 02 13:44:10.682048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.681938    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7f9986f-b8g5g"
Nov 02 13:44:10.722329 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.720977    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd7f9986f-b8g5g"]
Nov 02 13:44:10.725617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.725591    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dd7f9986f-b8g5g"]
Nov 02 13:44:10.965903 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:10.965825    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qbkw"
Nov 02 13:44:11.539574 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.539530    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" path="/var/lib/kubelet/pods/62b7284f-c8bf-4bd4-8b71-16b297e9d2e8/volumes"
Nov 02 13:44:11.688775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688744    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe" exitCode=0
Nov 02 13:44:11.688775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688767    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7" exitCode=0
Nov 02 13:44:11.688775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688775    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39" exitCode=0
Nov 02 13:44:11.688775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688781    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5" exitCode=0
Nov 02 13:44:11.689345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688788    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0" exitCode=0
Nov 02 13:44:11.689345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688815    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe"}
Nov 02 13:44:11.689345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688880    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7"}
Nov 02 13:44:11.689345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688912    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39"}
Nov 02 13:44:11.689345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688941    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5"}
Nov 02 13:44:11.689345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.688975    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0"}
Nov 02 13:44:11.893240 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.893215    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:11.994147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994116    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994148    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994184    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994226    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994256    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994285    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994316    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994346    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994375    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994400    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994906 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994437    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994906 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994470    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.994906 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.994520    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\") pod \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\" (UID: \"0c9eb4eb-569f-49eb-ac33-e670af0472a9\") "
Nov 02 13:44:11.995053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.995021    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:11.995622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.995544    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:11.995912 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.995827    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:11.998297 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.998262    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:11.998297 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.998275    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:11.998523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.998496    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:11.999156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.999122    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:11.999484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.999462    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:11.999610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.999481    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "kube-api-access-zs222". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:11.999769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.999747    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:11.999878 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:11.999833    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:12.002989 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.002898    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:12.008768 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.008743    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config" pod "0c9eb4eb-569f-49eb-ac33-e670af0472a9" (UID: "0c9eb4eb-569f-49eb-ac33-e670af0472a9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095827    5572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-main-db\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095859    5572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-tls-assets\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095869    5572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-metrics-client-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095877    5572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-out\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095887    5572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-web-config\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095896    5572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-cluster-tls-config\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095905    5572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-main-tls\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.095910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095914    5572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9eb4eb-569f-49eb-ac33-e670af0472a9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.096255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095923    5572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.096255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095932    5572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.096255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095940    5572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.096255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095950    5572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0c9eb4eb-569f-49eb-ac33-e670af0472a9-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.096255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.095958    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs222\" (UniqueName: \"kubernetes.io/projected/0c9eb4eb-569f-49eb-ac33-e670af0472a9-kube-api-access-zs222\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:44:12.696260 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.696224    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerID="98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c" exitCode=0
Nov 02 13:44:12.696709 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.696282    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c"}
Nov 02 13:44:12.696709 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.696308    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.696709 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.696336    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0c9eb4eb-569f-49eb-ac33-e670af0472a9","Type":"ContainerDied","Data":"2e94e799b19519e8be76d5d383263ba5b0eaf120953dd9ab47ea8ea9aa9c6b06"}
Nov 02 13:44:12.696709 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.696368    5572 scope.go:122] "RemoveContainer" containerID="1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe"
Nov 02 13:44:12.703706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.703680    5572 scope.go:122] "RemoveContainer" containerID="324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7"
Nov 02 13:44:12.710388 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.710370    5572 scope.go:122] "RemoveContainer" containerID="41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39"
Nov 02 13:44:12.717052 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.717037    5572 scope.go:122] "RemoveContainer" containerID="98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c"
Nov 02 13:44:12.723573 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.723544    5572 scope.go:122] "RemoveContainer" containerID="012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5"
Nov 02 13:44:12.724013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.723995    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:12.729487 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.729468    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:12.731533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.731509    5572 scope.go:122] "RemoveContainer" containerID="ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0"
Nov 02 13:44:12.738467 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.738449    5572 scope.go:122] "RemoveContainer" containerID="5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a"
Nov 02 13:44:12.744981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.744963    5572 scope.go:122] "RemoveContainer" containerID="1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe"
Nov 02 13:44:12.745242 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:12.745222    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe\": container with ID starting with 1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe not found: ID does not exist" containerID="1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe"
Nov 02 13:44:12.745300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.745251    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe"} err="failed to get container status \"1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe\": rpc error: code = NotFound desc = could not find container \"1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe\": container with ID starting with 1f3d2f1b98a5d26120290db3d66485d9811921eec83f4325243b71afa7913abe not found: ID does not exist"
Nov 02 13:44:12.745300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.745295    5572 scope.go:122] "RemoveContainer" containerID="324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7"
Nov 02 13:44:12.745589 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:12.745547    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7\": container with ID starting with 324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7 not found: ID does not exist" containerID="324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7"
Nov 02 13:44:12.745656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.745602    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7"} err="failed to get container status \"324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7\": rpc error: code = NotFound desc = could not find container \"324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7\": container with ID starting with 324f9d2471cdf4a36c44683074d86ae4b46484252f92ed439b6453982144f3d7 not found: ID does not exist"
Nov 02 13:44:12.745656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.745624    5572 scope.go:122] "RemoveContainer" containerID="41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39"
Nov 02 13:44:12.745900 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:12.745876    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39\": container with ID starting with 41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39 not found: ID does not exist" containerID="41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39"
Nov 02 13:44:12.745969 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.745903    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39"} err="failed to get container status \"41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39\": rpc error: code = NotFound desc = could not find container \"41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39\": container with ID starting with 41a4e72cd418f3b123d7db68641b5938ce13f79d8fa92c6e43e8781e5ca05f39 not found: ID does not exist"
Nov 02 13:44:12.745969 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.745923    5572 scope.go:122] "RemoveContainer" containerID="98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c"
Nov 02 13:44:12.746174 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:12.746156    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c\": container with ID starting with 98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c not found: ID does not exist" containerID="98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c"
Nov 02 13:44:12.746245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.746176    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c"} err="failed to get container status \"98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c\": rpc error: code = NotFound desc = could not find container \"98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c\": container with ID starting with 98aeba42fc841a22d0a5a71233377221810e24a046fe2e09b568be64f2ed357c not found: ID does not exist"
Nov 02 13:44:12.746245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.746192    5572 scope.go:122] "RemoveContainer" containerID="012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5"
Nov 02 13:44:12.746403 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:12.746386    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5\": container with ID starting with 012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5 not found: ID does not exist" containerID="012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5"
Nov 02 13:44:12.746442 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.746409    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5"} err="failed to get container status \"012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5\": rpc error: code = NotFound desc = could not find container \"012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5\": container with ID starting with 012b140e7705bb5b0c67964cf4c41316c6f0208b3af3f31252b892c248dd69b5 not found: ID does not exist"
Nov 02 13:44:12.746442 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.746424    5572 scope.go:122] "RemoveContainer" containerID="ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0"
Nov 02 13:44:12.746722 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:12.746704    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0\": container with ID starting with ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0 not found: ID does not exist" containerID="ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0"
Nov 02 13:44:12.746775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.746724    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0"} err="failed to get container status \"ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0\": rpc error: code = NotFound desc = could not find container \"ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0\": container with ID starting with ed4280a13e1abb3e521846a7fea32606cb1f38b8f30a48d281c6d2ee1efc93c0 not found: ID does not exist"
Nov 02 13:44:12.746775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.746736    5572 scope.go:122] "RemoveContainer" containerID="5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a"
Nov 02 13:44:12.746945 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:12.746927    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a\": container with ID starting with 5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a not found: ID does not exist" containerID="5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a"
Nov 02 13:44:12.747021 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.746947    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a"} err="failed to get container status \"5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a\": rpc error: code = NotFound desc = could not find container \"5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a\": container with ID starting with 5ef166e2bfcdbaa526f0051e4abefa22de4cac162ac6fc190d49ec98f15f136a not found: ID does not exist"
Nov 02 13:44:12.782037 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782006    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:12.782271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782259    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-metric"
Nov 02 13:44:12.782325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782272    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-metric"
Nov 02 13:44:12.782325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782283    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="alertmanager"
Nov 02 13:44:12.782325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782288    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="alertmanager"
Nov 02 13:44:12.782325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782301    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="prom-label-proxy"
Nov 02 13:44:12.782325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782307    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="prom-label-proxy"
Nov 02 13:44:12.782325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782317    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-web"
Nov 02 13:44:12.782325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782324    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-web"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782336    5572 cpu_manager.go:414] "containerMap: removing container" podUID="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" containerName="console"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782341    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" containerName="console"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782350    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="config-reloader"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782354    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="config-reloader"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782361    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="init-config-reloader"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782366    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="init-config-reloader"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782372    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782376    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782418    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="prom-label-proxy"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782425    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-web"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782430    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b7284f-c8bf-4bd4-8b71-16b297e9d2e8" containerName="console"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782435    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="config-reloader"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782440    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="init-config-reloader"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782445    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782452    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="kube-rbac-proxy-metric"
Nov 02 13:44:12.782533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.782457    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" containerName="alertmanager"
Nov 02 13:44:12.785927 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.785909    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.793176 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793148    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 13:44:12.793176 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793168    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 13:44:12.793411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793168    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 13:44:12.793411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793186    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 13:44:12.793411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793149    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 13:44:12.793722 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793456    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 13:44:12.793722 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793457    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 13:44:12.793722 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793512    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 13:44:12.793722 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793689    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 13:44:12.793933 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.793890    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f2hpb\""
Nov 02 13:44:12.806751 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.806665    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:12.879848 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.879825    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gbsrr"
Nov 02 13:44:12.901618 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901588    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.901772 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901628    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlkc\" (UniqueName: \"kubernetes.io/projected/719ae74f-7be3-4208-9462-d98255364534-kube-api-access-szlkc\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.901772 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901654    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.901772 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901765    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-web-config\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.901893 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901787    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/719ae74f-7be3-4208-9462-d98255364534-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.901893 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901809    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.901893 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901828    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/719ae74f-7be3-4208-9462-d98255364534-tls-assets\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.902002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901927    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-config-volume\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.902002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.901961    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.902078 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.902023    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.902128 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.902100    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/719ae74f-7be3-4208-9462-d98255364534-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.902168 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.902130    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/719ae74f-7be3-4208-9462-d98255364534-config-out\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:12.902168 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:12.902149    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/719ae74f-7be3-4208-9462-d98255364534-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002552    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002619    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/719ae74f-7be3-4208-9462-d98255364534-tls-assets\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002638    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-config-volume\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002656    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002674    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002703    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/719ae74f-7be3-4208-9462-d98255364534-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002720    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/719ae74f-7be3-4208-9462-d98255364534-config-out\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.002967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002739    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/719ae74f-7be3-4208-9462-d98255364534-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.003157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002961    5572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:44:13.003157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.002978    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.003157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.003011    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlkc\" (UniqueName: \"kubernetes.io/projected/719ae74f-7be3-4208-9462-d98255364534-kube-api-access-szlkc\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.003157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.003039    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.003157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.003076    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-web-config\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.003157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.003103    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/719ae74f-7be3-4208-9462-d98255364534-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.003157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.003152    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/719ae74f-7be3-4208-9462-d98255364534-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.003588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.003546    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/719ae74f-7be3-4208-9462-d98255364534-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.004232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.004191    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/719ae74f-7be3-4208-9462-d98255364534-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.006296 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.006270    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/719ae74f-7be3-4208-9462-d98255364534-tls-assets\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.006853 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.006443    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.006853 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.006675    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-config-volume\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.006853 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.006700    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/719ae74f-7be3-4208-9462-d98255364534-config-out\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.006853 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.006804    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.006853 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.006849    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.007053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.006880    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.007178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.007157    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-web-config\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.007788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.007771    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/719ae74f-7be3-4208-9462-d98255364534-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.018246 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.018215    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlkc\" (UniqueName: \"kubernetes.io/projected/719ae74f-7be3-4208-9462-d98255364534-kube-api-access-szlkc\") pod \"alertmanager-main-0\" (UID: \"719ae74f-7be3-4208-9462-d98255364534\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.104247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.104202    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:13.392128 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.392086    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:13.396011 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:44:13.395982    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719ae74f_7be3_4208_9462_d98255364534.slice/crio-e3647e7e2613ef9f04120d151f9023cd420c1d9fc2a0458b67a4c8cb5f193ac9 WatchSource:0}: Error finding container e3647e7e2613ef9f04120d151f9023cd420c1d9fc2a0458b67a4c8cb5f193ac9: Status 404 returned error can't find the container with id e3647e7e2613ef9f04120d151f9023cd420c1d9fc2a0458b67a4c8cb5f193ac9
Nov 02 13:44:13.542474 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.542444    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9eb4eb-569f-49eb-ac33-e670af0472a9" path="/var/lib/kubelet/pods/0c9eb4eb-569f-49eb-ac33-e670af0472a9/volumes"
Nov 02 13:44:13.701834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.701736    5572 generic.go:356] "Generic (PLEG): container finished" podID="719ae74f-7be3-4208-9462-d98255364534" containerID="355b10b2fb931728340a553af250e9a78b4831695101c69274f34fa7f42b768f" exitCode=0
Nov 02 13:44:13.701834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.701782    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerDied","Data":"355b10b2fb931728340a553af250e9a78b4831695101c69274f34fa7f42b768f"}
Nov 02 13:44:13.701834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:13.701833    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerStarted","Data":"e3647e7e2613ef9f04120d151f9023cd420c1d9fc2a0458b67a4c8cb5f193ac9"}
Nov 02 13:44:14.664709 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.664671    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85fd9b44-djhqk"]
Nov 02 13:44:14.669052 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.669030    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.689337 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.689310    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85fd9b44-djhqk"]
Nov 02 13:44:14.709212 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.709177    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerStarted","Data":"8c6efdd04b1d422a3435ec306833d59fdd7775e68efe29c4a8f4d0b892533edf"}
Nov 02 13:44:14.709589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.709241    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerStarted","Data":"d0de14224c05b24125c6e8cb9f40a363368c7bdced5443ce4137e0cd686e29a6"}
Nov 02 13:44:14.709589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.709272    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerStarted","Data":"4fc0cd928cdd4ce26a15f5d56d506a0d7f361b349b080be1facf57460da06328"}
Nov 02 13:44:14.709589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.709301    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerStarted","Data":"71bbc08d6d35871e2cec77cb3728247e4aa7dd2f4282b81151de6d10161bf4ab"}
Nov 02 13:44:14.709589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.709330    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerStarted","Data":"628f147c7da281bfb89456e104c619644e91b15bb827172aaeba56d219b367cd"}
Nov 02 13:44:14.709589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.709355    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719ae74f-7be3-4208-9462-d98255364534","Type":"ContainerStarted","Data":"afc481d140a93b51cabb4b4dd8a433462a845053f82094cce841ab291a42a6cc"}
Nov 02 13:44:14.717272 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717249    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-federate-client-tls\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.717408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717308    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.717408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717372    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4tk\" (UniqueName: \"kubernetes.io/projected/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-kube-api-access-nz4tk\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.717538 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717406    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-serving-certs-ca-bundle\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.717538 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717470    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-metrics-client-ca\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.717680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717620    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-secret-telemeter-client\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.717680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717650    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-telemeter-client-tls\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.717779 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.717694    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.753489 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.753455    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 13:44:14.753662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.753645    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 13:44:14.753761 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.753687    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 13:44:14.753897 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.753885    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z9fr5\""
Nov 02 13:44:14.753943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.753924    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 13:44:14.756576 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.756546    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 13:44:14.758739 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.758723    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 13:44:14.763252 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.763212    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.7511736989999998 podStartE2EDuration="2.763199976s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.026277ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:14.761486677 +0000 UTC m=+31.339575545" watchObservedRunningTime="2026-11-02 13:44:14.763199976 +0000 UTC m=+31.341288840"
Nov 02 13:44:14.819046 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.819015    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-secret-telemeter-client\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.819046 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.819049    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-telemeter-client-tls\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.819300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.819072    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.819300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.819171    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-federate-client-tls\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.819551 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.819521    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.819679 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.819591    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4tk\" (UniqueName: \"kubernetes.io/projected/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-kube-api-access-nz4tk\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.819985 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.819963    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.820289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.820264    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-serving-certs-ca-bundle\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.820384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.820303    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-metrics-client-ca\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.821201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.821178    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-metrics-client-ca\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.821282 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.821224    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-serving-certs-ca-bundle\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.822197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.822157    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-federate-client-tls\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.822343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.822322    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-telemeter-client-tls\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.822993 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.822948    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.825366 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.825336    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-secret-telemeter-client\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.834601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.834544    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4tk\" (UniqueName: \"kubernetes.io/projected/eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b-kube-api-access-nz4tk\") pod \"telemeter-client-85fd9b44-djhqk\" (UID: \"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b\") " pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:14.989723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:14.989647    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk"
Nov 02 13:44:15.293684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:15.293594    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85fd9b44-djhqk"]
Nov 02 13:44:15.297047 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:44:15.297017    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9d31b8_c96d_44d0_a6c2_ebaf72e0577b.slice/crio-e7a41e6e06c5d53925258c8977153d2f564250b6ac8a99db2fec473b56e6ab20 WatchSource:0}: Error finding container e7a41e6e06c5d53925258c8977153d2f564250b6ac8a99db2fec473b56e6ab20: Status 404 returned error can't find the container with id e7a41e6e06c5d53925258c8977153d2f564250b6ac8a99db2fec473b56e6ab20
Nov 02 13:44:15.299758 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:15.299730    5572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:44:15.713853 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:15.713814    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk" event={"ID":"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b","Type":"ContainerStarted","Data":"e7a41e6e06c5d53925258c8977153d2f564250b6ac8a99db2fec473b56e6ab20"}
Nov 02 13:44:17.722545 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.722512    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk" event={"ID":"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b","Type":"ContainerStarted","Data":"fda99fb369aa6c62985eb5d3527ebcb71b4d7a3b29a99df21e491e8f21c00e45"}
Nov 02 13:44:17.723016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.722588    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk" event={"ID":"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b","Type":"ContainerStarted","Data":"7888b1762fc0e11b039478972f60995e253973a6c52bf698e5517d5715377de5"}
Nov 02 13:44:17.723016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.722615    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk" event={"ID":"eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b","Type":"ContainerStarted","Data":"1717536e38e4007df86a38bf2a087777129a00ea1be2a55efa67f18047ef6ef9"}
Nov 02 13:44:17.752520 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.752468    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85fd9b44-djhqk" podStartSLOduration=2.266314192 podStartE2EDuration="3.752455186s" totalImagesPullingTime="1.486140994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:17.752017989 +0000 UTC m=+34.330106869" watchObservedRunningTime="2026-11-02 13:44:17.752455186 +0000 UTC m=+34.330544054"
Nov 02 13:44:17.860154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.860128    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76d8f6d577-x76df"]
Nov 02 13:44:17.862182 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.862162    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.889966 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.889943    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d8f6d577-x76df"]
Nov 02 13:44:17.894323 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894307    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 13:44:17.894553 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894535    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 13:44:17.894680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894590    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 13:44:17.894680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894615    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mtv77\""
Nov 02 13:44:17.894680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894618    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 13:44:17.894680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894664    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 13:44:17.894857 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894768    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 13:44:17.894857 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.894783    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 13:44:17.897514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.897497    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 13:44:17.950516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950485    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvv96\" (UniqueName: \"kubernetes.io/projected/3ef91520-0e36-4dae-bee0-c8d7da74984a-kube-api-access-zvv96\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.950516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950516    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-trusted-ca-bundle\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.950726 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950544    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-service-ca\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.950726 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950693    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-config\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.950824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950726    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef91520-0e36-4dae-bee0-c8d7da74984a-tmp\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.950824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950811    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-oauth-serving-cert\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.950898 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950837    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-serving-cert\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:17.950898 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:17.950855    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-oauth-config\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052200 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052120    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-config\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052200 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052154    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef91520-0e36-4dae-bee0-c8d7da74984a-tmp\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052200 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052183    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-oauth-serving-cert\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052318    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-serving-cert\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052349    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-oauth-config\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052397    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvv96\" (UniqueName: \"kubernetes.io/projected/3ef91520-0e36-4dae-bee0-c8d7da74984a-kube-api-access-zvv96\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052426    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-trusted-ca-bundle\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052668 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052479    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-service-ca\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052699    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef91520-0e36-4dae-bee0-c8d7da74984a-tmp\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.052993 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052970    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-config\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.053088 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.052989    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-oauth-serving-cert\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.053226 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.053204    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-service-ca\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.053312 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.053251    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-trusted-ca-bundle\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.055466 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.055446    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-serving-cert\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.055571 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.055539    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-oauth-config\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.065441 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.065418    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvv96\" (UniqueName: \"kubernetes.io/projected/3ef91520-0e36-4dae-bee0-c8d7da74984a-kube-api-access-zvv96\") pod \"console-76d8f6d577-x76df\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") " pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.172190 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.172141    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:18.444896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.444867    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d8f6d577-x76df"]
Nov 02 13:44:18.448118 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:44:18.448079    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef91520_0e36_4dae_bee0_c8d7da74984a.slice/crio-79d2a65058bd07c77cdba624c19d7dcfd5ff3116f9f0669a9945db1f649147b4 WatchSource:0}: Error finding container 79d2a65058bd07c77cdba624c19d7dcfd5ff3116f9f0669a9945db1f649147b4: Status 404 returned error can't find the container with id 79d2a65058bd07c77cdba624c19d7dcfd5ff3116f9f0669a9945db1f649147b4
Nov 02 13:44:18.729014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.728932    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d8f6d577-x76df" event={"ID":"3ef91520-0e36-4dae-bee0-c8d7da74984a","Type":"ContainerStarted","Data":"08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6"}
Nov 02 13:44:18.729014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.728981    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d8f6d577-x76df" event={"ID":"3ef91520-0e36-4dae-bee0-c8d7da74984a","Type":"ContainerStarted","Data":"79d2a65058bd07c77cdba624c19d7dcfd5ff3116f9f0669a9945db1f649147b4"}
Nov 02 13:44:18.752922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:18.752869    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76d8f6d577-x76df" podStartSLOduration=1.752855158 podStartE2EDuration="1.752855158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:18.752257039 +0000 UTC m=+35.330345910" watchObservedRunningTime="2026-11-02 13:44:18.752855158 +0000 UTC m=+35.330944024"
Nov 02 13:44:21.428767 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:21.428731    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:44:28.173282 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:28.173250    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:28.173691 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:28.173310    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:28.178263 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:28.178244    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:28.765887 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:44:28.765860    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:44:51.427875 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:44:51.427839    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:45:15.678350 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.678321    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd84cb498-hktk5"]
Nov 02 13:45:15.681608 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.681590    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.699020 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.698987    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd84cb498-hktk5"]
Nov 02 13:45:15.773932 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.773893    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-config\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.774107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.773943    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-trusted-ca-bundle\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.774107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.774012    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-oauth-config\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.774107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.774082    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-oauth-serving-cert\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.774107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.774106    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-serving-cert\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.774293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.774178    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-tmp\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.774293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.774206    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-service-ca\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.774293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.774230    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwtk\" (UniqueName: \"kubernetes.io/projected/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-kube-api-access-2dwtk\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.874845 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.874808    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-config\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.874845 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.874851    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-trusted-ca-bundle\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.874874    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-oauth-config\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875006    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-oauth-serving-cert\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875048    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-serving-cert\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875228 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875178    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-tmp\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875286 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875229    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-service-ca\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875286 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875271    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwtk\" (UniqueName: \"kubernetes.io/projected/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-kube-api-access-2dwtk\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875717 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875695    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-config\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875803    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-tmp\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875950 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875932    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-service-ca\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.875988 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.875971    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-oauth-serving-cert\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.876593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.876552    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-trusted-ca-bundle\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.877394 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.877371    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-oauth-config\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.877616 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.877601    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-console-serving-cert\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.888293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.888263    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwtk\" (UniqueName: \"kubernetes.io/projected/7ce3b8ce-ae26-4d13-a669-b511ecbd75f6-kube-api-access-2dwtk\") pod \"console-cd84cb498-hktk5\" (UID: \"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6\") " pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:15.992793 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:15.992689    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:16.270388 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:16.270318    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd84cb498-hktk5"]
Nov 02 13:45:16.273621 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:45:16.273598    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce3b8ce_ae26_4d13_a669_b511ecbd75f6.slice/crio-ac44a515ace23ee1f8b485c0429b7244bc1b1a284949a75cdb92ac7aa0baee36 WatchSource:0}: Error finding container ac44a515ace23ee1f8b485c0429b7244bc1b1a284949a75cdb92ac7aa0baee36: Status 404 returned error can't find the container with id ac44a515ace23ee1f8b485c0429b7244bc1b1a284949a75cdb92ac7aa0baee36
Nov 02 13:45:16.924681 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:16.924645    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd84cb498-hktk5" event={"ID":"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6","Type":"ContainerStarted","Data":"5167f542459869b107dd51c1d6f51733a41f526259fe5c8d630bfb7a643aa645"}
Nov 02 13:45:16.925102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:16.924734    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd84cb498-hktk5" event={"ID":"7ce3b8ce-ae26-4d13-a669-b511ecbd75f6","Type":"ContainerStarted","Data":"ac44a515ace23ee1f8b485c0429b7244bc1b1a284949a75cdb92ac7aa0baee36"}
Nov 02 13:45:16.949405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:16.949359    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cd84cb498-hktk5" podStartSLOduration=1.949347479 podStartE2EDuration="1.949347479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:45:16.947885421 +0000 UTC m=+93.525974285" watchObservedRunningTime="2026-11-02 13:45:16.949347479 +0000 UTC m=+93.527436345"
Nov 02 13:45:21.428697 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:45:21.428659    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:45:25.993738 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:25.993638    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:25.993738 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:25.993710    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:25.998432 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:25.998410    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:26.963377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:26.963351    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd84cb498-hktk5"
Nov 02 13:45:27.019400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:27.019370    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d8f6d577-x76df"]
Nov 02 13:45:51.428782 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:45:51.428743    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:45:52.039254 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.039197    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76d8f6d577-x76df" podUID="3ef91520-0e36-4dae-bee0-c8d7da74984a" containerName="console" containerID="cri-o://08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6" gracePeriod=15
Nov 02 13:45:52.271281 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.271259    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:45:52.282776 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.282753    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef91520-0e36-4dae-bee0-c8d7da74984a-kube-api-access-zvv96\" (UniqueName: \"kubernetes.io/projected/3ef91520-0e36-4dae-bee0-c8d7da74984a-kube-api-access-zvv96\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.282902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.282788    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-oauth-config\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.282902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.282835    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-service-ca\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.282902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.282866    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-oauth-serving-cert\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.283066 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.282913    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-trusted-ca-bundle\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.283066 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.282941    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-config\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-config\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.283066 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.282969    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-serving-cert\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.283066 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.283021    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef91520-0e36-4dae-bee0-c8d7da74984a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef91520-0e36-4dae-bee0-c8d7da74984a-tmp\") pod \"3ef91520-0e36-4dae-bee0-c8d7da74984a\" (UID: \"3ef91520-0e36-4dae-bee0-c8d7da74984a\") "
Nov 02 13:45:52.283307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.283280    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-service-ca" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:52.283380 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.283284    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-oauth-serving-cert" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:52.283380 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.283314    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-trusted-ca-bundle" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:52.283514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.283495    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef91520-0e36-4dae-bee0-c8d7da74984a-tmp" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:52.283757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.283728    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-config" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:52.285046 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.285012    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef91520-0e36-4dae-bee0-c8d7da74984a-kube-api-access-zvv96" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "kube-api-access-zvv96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:52.285137 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.285059    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-oauth-config" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:52.285459 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.285430    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-serving-cert" pod "3ef91520-0e36-4dae-bee0-c8d7da74984a" (UID: "3ef91520-0e36-4dae-bee0-c8d7da74984a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:52.383936 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383912    5572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-trusted-ca-bundle\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:52.383936 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383933    5572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-config\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:52.384111 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383942    5572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-serving-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:52.384111 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383951    5572 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef91520-0e36-4dae-bee0-c8d7da74984a-tmp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:52.384111 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383960    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvv96\" (UniqueName: \"kubernetes.io/projected/3ef91520-0e36-4dae-bee0-c8d7da74984a-kube-api-access-zvv96\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:52.384111 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383970    5572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef91520-0e36-4dae-bee0-c8d7da74984a-console-oauth-config\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:52.384111 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383978    5572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-service-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:52.384111 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:52.383986    5572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef91520-0e36-4dae-bee0-c8d7da74984a-oauth-serving-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:45:53.045897 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.045864    5572 generic.go:356] "Generic (PLEG): container finished" podID="3ef91520-0e36-4dae-bee0-c8d7da74984a" containerID="08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6" exitCode=0
Nov 02 13:45:53.046317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.045906    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d8f6d577-x76df"
Nov 02 13:45:53.046317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.045954    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d8f6d577-x76df" event={"ID":"3ef91520-0e36-4dae-bee0-c8d7da74984a","Type":"ContainerDied","Data":"08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6"}
Nov 02 13:45:53.046317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.046006    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d8f6d577-x76df" event={"ID":"3ef91520-0e36-4dae-bee0-c8d7da74984a","Type":"ContainerDied","Data":"79d2a65058bd07c77cdba624c19d7dcfd5ff3116f9f0669a9945db1f649147b4"}
Nov 02 13:45:53.046317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.046025    5572 scope.go:122] "RemoveContainer" containerID="08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6"
Nov 02 13:45:53.054680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.054666    5572 scope.go:122] "RemoveContainer" containerID="08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6"
Nov 02 13:45:53.054935 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:45:53.054916    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6\": container with ID starting with 08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6 not found: ID does not exist" containerID="08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6"
Nov 02 13:45:53.055000 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.054942    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6"} err="failed to get container status \"08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6\": rpc error: code = NotFound desc = could not find container \"08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6\": container with ID starting with 08a5633a866e8ac3ab2de25d7a76be11b24c375036481444ef4d7a2d6de03dd6 not found: ID does not exist"
Nov 02 13:45:53.070939 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.070917    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d8f6d577-x76df"]
Nov 02 13:45:53.078073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.078049    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76d8f6d577-x76df"]
Nov 02 13:45:53.542784 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:45:53.541167    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef91520-0e36-4dae-bee0-c8d7da74984a" path="/var/lib/kubelet/pods/3ef91520-0e36-4dae-bee0-c8d7da74984a/volumes"
Nov 02 13:46:21.428372 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:46:21.428333    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:46:51.429079 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:46:51.429037    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:47:21.428658 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:21.428622    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:47:51.429113 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:51.429077    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:47:55.474144 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.474113    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"]
Nov 02 13:47:55.474623 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.474388    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3ef91520-0e36-4dae-bee0-c8d7da74984a" containerName="console"
Nov 02 13:47:55.474623 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.474398    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef91520-0e36-4dae-bee0-c8d7da74984a" containerName="console"
Nov 02 13:47:55.474623 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.474460    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef91520-0e36-4dae-bee0-c8d7da74984a" containerName="console"
Nov 02 13:47:55.477072 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.477057    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.487070 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:55.487040    5572 status_manager.go:1045] "Failed to get status for pod" err="pods \"istiod-openshift-gateway-7876bfcb74-lf6qh\" is forbidden: User \"system:node:ip-10-0-128-62.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-62.us-east-2.compute.internal' and this object" podUID="2a156c79-3139-41f1-b2fe-cf5ac007d864" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.506023 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:55.505982    5572 reflector.go:204] "Failed to watch" err="failed to list *v1.Secret: secrets \"istio-kubeconfig\" is forbidden: User \"system:node:ip-10-0-128-62.us-east-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-62.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret"
Nov 02 13:47:55.506173 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:55.506098    5572 reflector.go:204] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-128-62.us-east-2.compute.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-62.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap"
Nov 02 13:47:55.506235 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:55.506212    5572 reflector.go:204] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-openshift-gateway-dockercfg-kmsk4\" is forbidden: User \"system:node:ip-10-0-128-62.us-east-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-62.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kmsk4\"" type="*v1.Secret"
Nov 02 13:47:55.506282 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:55.506227    5572 reflector.go:204] "Failed to watch" err="failed to list *v1.Secret: secrets \"cacerts\" is forbidden: User \"system:node:ip-10-0-128-62.us-east-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-62.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret"
Nov 02 13:47:55.512409 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.512392    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 13:47:55.515823 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.515810    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 13:47:55.516945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.516932    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 13:47:55.601509 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.601488    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"]
Nov 02 13:47:55.626926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.626901    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwdp\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-kube-api-access-7cwdp\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.627026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.626946    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.627026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.626971    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.627105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.627059    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.627105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.627090    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.627197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.627109    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a156c79-3139-41f1-b2fe-cf5ac007d864-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.627197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.627125    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.728388 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.728324    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.728388 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.728359    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.728587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.728389    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.728587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.728418    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.728587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.728505    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a156c79-3139-41f1-b2fe-cf5ac007d864-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.728587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.728535    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.728794 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.728602    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwdp\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-kube-api-access-7cwdp\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.730820 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.730794    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a156c79-3139-41f1-b2fe-cf5ac007d864-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.730973 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.730957    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.741821 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.741800    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:55.743069 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:55.743048    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwdp\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-kube-api-access-7cwdp\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:56.372692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:56.372660    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 13:47:56.380951 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:56.380925    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:56.513832 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:56.513805    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 13:47:56.521064 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:56.521043    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:56.728938 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:56.728874    5572 configmap.go:193] Couldn't get configMap openshift-ingress/openshift-gw-ca-root-cert: failed to sync configmap cache: timed out waiting for the condition
Nov 02 13:47:56.729066 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:47:56.728951    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap podName:2a156c79-3139-41f1-b2fe-cf5ac007d864 nodeName:}" failed. No retries permitted until 2026-11-02 13:47:57.22893394 +0000 UTC m=+253.807022798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istio-csr-ca-configmap" (UniqueName: "kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap") pod "istiod-openshift-gateway-7876bfcb74-lf6qh" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864") : failed to sync configmap cache: timed out waiting for the condition
Nov 02 13:47:56.896362 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:56.896334    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 13:47:57.100987 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:57.100957    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kmsk4\""
Nov 02 13:47:57.242123 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:57.242093    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:57.242631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:57.242614    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lf6qh\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:57.288021 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:57.287999    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:47:57.563163 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:57.563141    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"]
Nov 02 13:47:57.566466 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:47:57.566442    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a156c79_3139_41f1_b2fe_cf5ac007d864.slice/crio-63cedeb3f41e4e7286a36faf5320b1651188424c8b798b7dc1e20d998d141a27 WatchSource:0}: Error finding container 63cedeb3f41e4e7286a36faf5320b1651188424c8b798b7dc1e20d998d141a27: Status 404 returned error can't find the container with id 63cedeb3f41e4e7286a36faf5320b1651188424c8b798b7dc1e20d998d141a27
Nov 02 13:47:58.468674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:47:58.468634    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh" event={"ID":"2a156c79-3139-41f1-b2fe-cf5ac007d864","Type":"ContainerStarted","Data":"63cedeb3f41e4e7286a36faf5320b1651188424c8b798b7dc1e20d998d141a27"}
Nov 02 13:48:00.227792 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:00.227762    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:48:00.228069 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:00.227820    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:48:00.478763 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:00.478682    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh" event={"ID":"2a156c79-3139-41f1-b2fe-cf5ac007d864","Type":"ContainerStarted","Data":"40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13"}
Nov 02 13:48:00.478917 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:00.478814    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:48:00.500675 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:00.500629    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh" podStartSLOduration=2.841958222 podStartE2EDuration="5.500618341s" totalImagesPullingTime="2.658660119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:00.499125986 +0000 UTC m=+257.077214852" watchObservedRunningTime="2026-11-02 13:48:00.500618341 +0000 UTC m=+257.078707208"
Nov 02 13:48:01.483703 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:01.483664    5572 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lf6qh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:48:01.484164 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:01.483739    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh" podUID="2a156c79-3139-41f1-b2fe-cf5ac007d864" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:48:04.483290 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:04.483264    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:48:21.428274 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:48:21.428240    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:48:45.328671 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.328586    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"]
Nov 02 13:48:45.338402 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.338372    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.363476 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.363442    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"]
Nov 02 13:48:45.386665 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.386638    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.386783 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.386678    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hn8q\" (UniqueName: \"kubernetes.io/projected/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-kube-api-access-7hn8q\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.386827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.386791    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-metrics-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.386864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.386839    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.386864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.386857    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-tmp\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.386923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.386888    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-manifests\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.402319 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.402299    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 13:48:45.402319 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.402312    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-qdqbf\""
Nov 02 13:48:45.402472 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.402394    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 13:48:45.402555 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.402542    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 13:48:45.402620 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.402590    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 13:48:45.402655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.402630    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 13:48:45.487502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.487480    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.487699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.487517    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hn8q\" (UniqueName: \"kubernetes.io/projected/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-kube-api-access-7hn8q\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.487699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.487549    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-metrics-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.487699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.487594    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.487699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.487612    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-tmp\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.487699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.487651    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-manifests\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.488053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.488035    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-manifests\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.490077 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.490053    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-tmp\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.490179 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.490115    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.490440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.490423    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-metrics-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.490627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.490607    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.527437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.527408    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hn8q\" (UniqueName: \"kubernetes.io/projected/1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc-kube-api-access-7hn8q\") pod \"opendatahub-operator-controller-manager-c559bd947-hztw5\" (UID: \"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.653067 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.652999    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:45.967984 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:45.967913    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"]
Nov 02 13:48:45.974410 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:48:45.974386    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a716fcb_ad2a_41ed_8a1b_aadcdc9491bc.slice/crio-bf4700e28876162d228dac8c32f84c7326d8a1e6ebc76e185db09cc687ad3a20 WatchSource:0}: Error finding container bf4700e28876162d228dac8c32f84c7326d8a1e6ebc76e185db09cc687ad3a20: Status 404 returned error can't find the container with id bf4700e28876162d228dac8c32f84c7326d8a1e6ebc76e185db09cc687ad3a20
Nov 02 13:48:46.631302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:46.631267    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5" event={"ID":"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc","Type":"ContainerStarted","Data":"bf4700e28876162d228dac8c32f84c7326d8a1e6ebc76e185db09cc687ad3a20"}
Nov 02 13:48:49.642893 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:49.642864    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5" event={"ID":"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc","Type":"ContainerStarted","Data":"61ead57cbd2f0911e66d8816faa115c2d6d6a931a13d27bcc5f22080de2093db"}
Nov 02 13:48:50.648735 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:50.648701    5572 generic.go:356] "Generic (PLEG): container finished" podID="1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc" containerID="61ead57cbd2f0911e66d8816faa115c2d6d6a931a13d27bcc5f22080de2093db" exitCode=0
Nov 02 13:48:50.649183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:50.648784    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5" event={"ID":"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc","Type":"ContainerDied","Data":"61ead57cbd2f0911e66d8816faa115c2d6d6a931a13d27bcc5f22080de2093db"}
Nov 02 13:48:51.429119 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:48:51.429085    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:48:51.653774 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:51.653745    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5" event={"ID":"1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc","Type":"ContainerStarted","Data":"c05b940735854aa0ecbe0e5eae71672b465b18f5bc8ef6a300e61aa2c2cd35c5"}
Nov 02 13:48:51.654125 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:51.653851    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:48:51.700190 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:48:51.700113    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5" podStartSLOduration=2.250864309 podStartE2EDuration="6.700101408s" totalImagesPullingTime="3.989775385s" totalInitContainerRuntime="459.461714ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:51.69637372 +0000 UTC m=+308.274462587" watchObservedRunningTime="2026-11-02 13:48:51.700101408 +0000 UTC m=+308.278190266"
Nov 02 13:49:02.659133 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:02.659102    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c559bd947-hztw5"
Nov 02 13:49:16.685234 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.685202    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"]
Nov 02 13:49:16.690103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.690089    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.733422 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.733394    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"]
Nov 02 13:49:16.734518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.734497    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 13:49:16.734862 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.734839    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 13:49:16.734951 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.734839    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8jv2k\""
Nov 02 13:49:16.836281 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.836254    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cef31c7-8dba-4891-940f-1ef1d998854b-tmp\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.836455 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.836294    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cef31c7-8dba-4891-940f-1ef1d998854b-tls-certs\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.836455 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.836382    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdlz\" (UniqueName: \"kubernetes.io/projected/1cef31c7-8dba-4891-940f-1ef1d998854b-kube-api-access-wtdlz\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.936855 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.936794    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdlz\" (UniqueName: \"kubernetes.io/projected/1cef31c7-8dba-4891-940f-1ef1d998854b-kube-api-access-wtdlz\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.936855 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.936838    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cef31c7-8dba-4891-940f-1ef1d998854b-tmp\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.937070 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.936867    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cef31c7-8dba-4891-940f-1ef1d998854b-tls-certs\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.939610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.939585    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cef31c7-8dba-4891-940f-1ef1d998854b-tmp\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.939829 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.939808    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cef31c7-8dba-4891-940f-1ef1d998854b-tls-certs\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:16.976439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:16.976410    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdlz\" (UniqueName: \"kubernetes.io/projected/1cef31c7-8dba-4891-940f-1ef1d998854b-kube-api-access-wtdlz\") pod \"kube-auth-proxy-fb985f76c-7xxff\" (UID: \"1cef31c7-8dba-4891-940f-1ef1d998854b\") " pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:17.001244 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.001220    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"
Nov 02 13:49:17.314884 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.314859    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fb985f76c-7xxff"]
Nov 02 13:49:17.343323 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:49:17.343299    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cef31c7_8dba_4891_940f_1ef1d998854b.slice/crio-a4097763a51085b79cbf4e4772ed099ac064dedf1bf804a70c88a5edeebca662 WatchSource:0}: Error finding container a4097763a51085b79cbf4e4772ed099ac064dedf1bf804a70c88a5edeebca662: Status 404 returned error can't find the container with id a4097763a51085b79cbf4e4772ed099ac064dedf1bf804a70c88a5edeebca662
Nov 02 13:49:17.345194 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.345168    5572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:49:17.687913 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.687842    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"]
Nov 02 13:49:17.692153 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.692137    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:17.707279 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.707251    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"]
Nov 02 13:49:17.709096 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.709070    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 13:49:17.709242 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.709189    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 13:49:17.710373 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.710356    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 13:49:17.712006 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.711990    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6hrt4\""
Nov 02 13:49:17.741806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.741786    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff" event={"ID":"1cef31c7-8dba-4891-940f-1ef1d998854b","Type":"ContainerStarted","Data":"a4097763a51085b79cbf4e4772ed099ac064dedf1bf804a70c88a5edeebca662"}
Nov 02 13:49:17.782512 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.782486    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"]
Nov 02 13:49:17.844313 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.844289    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhj7m\" (UniqueName: \"kubernetes.io/projected/c4601a48-fa26-42af-92b3-3a482313aebb-kube-api-access-hhj7m\") pod \"limitador-limitador-74dfcfb894-vhb9k\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:17.844442 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.844329    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4601a48-fa26-42af-92b3-3a482313aebb-config-file\") pod \"limitador-limitador-74dfcfb894-vhb9k\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:17.945884 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.945820    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4601a48-fa26-42af-92b3-3a482313aebb-config-file\") pod \"limitador-limitador-74dfcfb894-vhb9k\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:17.946076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.945942    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhj7m\" (UniqueName: \"kubernetes.io/projected/c4601a48-fa26-42af-92b3-3a482313aebb-kube-api-access-hhj7m\") pod \"limitador-limitador-74dfcfb894-vhb9k\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:17.946678 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.946657    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4601a48-fa26-42af-92b3-3a482313aebb-config-file\") pod \"limitador-limitador-74dfcfb894-vhb9k\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:17.959645 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:17.959618    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhj7m\" (UniqueName: \"kubernetes.io/projected/c4601a48-fa26-42af-92b3-3a482313aebb-kube-api-access-hhj7m\") pod \"limitador-limitador-74dfcfb894-vhb9k\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:18.004889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:18.004863    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:18.303302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:18.303238    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"]
Nov 02 13:49:18.307706 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:49:18.307678    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4601a48_fa26_42af_92b3_3a482313aebb.slice/crio-406c2513ad5ac005fd2fea619e8eaf70dec5ba6dc9041264f5c26f874355cb87 WatchSource:0}: Error finding container 406c2513ad5ac005fd2fea619e8eaf70dec5ba6dc9041264f5c26f874355cb87: Status 404 returned error can't find the container with id 406c2513ad5ac005fd2fea619e8eaf70dec5ba6dc9041264f5c26f874355cb87
Nov 02 13:49:18.747267 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:18.747177    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k" event={"ID":"c4601a48-fa26-42af-92b3-3a482313aebb","Type":"ContainerStarted","Data":"406c2513ad5ac005fd2fea619e8eaf70dec5ba6dc9041264f5c26f874355cb87"}
Nov 02 13:49:20.662456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.662424    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"]
Nov 02 13:49:20.674486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.674458    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.683162 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.683027    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"]
Nov 02 13:49:20.685102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.685078    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-4w7fc\""
Nov 02 13:49:20.693530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.693480    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 13:49:20.693658 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.693535    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 13:49:20.697164 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.697142    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 13:49:20.757611 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.757550    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff" event={"ID":"1cef31c7-8dba-4891-940f-1ef1d998854b","Type":"ContainerStarted","Data":"b91c325430391caa57eda08fb25d2600e7c6067aa84a5d2cbece208e34373180"}
Nov 02 13:49:20.772078 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.772055    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c20ef56-0297-49b4-a716-f03385219dc0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.772197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.772102    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c20ef56-0297-49b4-a716-f03385219dc0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.772268 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.772220    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2ft\" (UniqueName: \"kubernetes.io/projected/0c20ef56-0297-49b4-a716-f03385219dc0-kube-api-access-5j2ft\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.772331 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.772301    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c20ef56-0297-49b4-a716-f03385219dc0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.873628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.873585    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c20ef56-0297-49b4-a716-f03385219dc0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.873796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.873727    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c20ef56-0297-49b4-a716-f03385219dc0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.873796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.873766    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c20ef56-0297-49b4-a716-f03385219dc0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.873796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.873788    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2ft\" (UniqueName: \"kubernetes.io/projected/0c20ef56-0297-49b4-a716-f03385219dc0-kube-api-access-5j2ft\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.874309 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.874257    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c20ef56-0297-49b4-a716-f03385219dc0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.874465 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.874337    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c20ef56-0297-49b4-a716-f03385219dc0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.874964 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.874945    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c20ef56-0297-49b4-a716-f03385219dc0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.888301 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.888275    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2ft\" (UniqueName: \"kubernetes.io/projected/0c20ef56-0297-49b4-a716-f03385219dc0-kube-api-access-5j2ft\") pod \"kserve-module-controller-manager-7566cf5fc8-vt89d\" (UID: \"0c20ef56-0297-49b4-a716-f03385219dc0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:20.990287 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:20.990265    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:21.281546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:21.281446    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-fb985f76c-7xxff" podStartSLOduration=2.801303592 podStartE2EDuration="5.281434107s" totalImagesPullingTime="2.480130515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:20.780881064 +0000 UTC m=+337.358969932" watchObservedRunningTime="2026-11-02 13:49:21.281434107 +0000 UTC m=+337.859522973"
Nov 02 13:49:21.282277 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:21.282253    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"]
Nov 02 13:49:21.287291 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:49:21.287266    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c20ef56_0297_49b4_a716_f03385219dc0.slice/crio-e8d113062d7c77a525bc5bb00f97c97d94163b3990bebbe2b8c6f973b964df1f WatchSource:0}: Error finding container e8d113062d7c77a525bc5bb00f97c97d94163b3990bebbe2b8c6f973b964df1f: Status 404 returned error can't find the container with id e8d113062d7c77a525bc5bb00f97c97d94163b3990bebbe2b8c6f973b964df1f
Nov 02 13:49:21.428417 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:49:21.428385    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:49:21.762475 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:21.762444    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k" event={"ID":"c4601a48-fa26-42af-92b3-3a482313aebb","Type":"ContainerStarted","Data":"545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0"}
Nov 02 13:49:21.762945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:21.762585    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:21.763735 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:21.763712    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d" event={"ID":"0c20ef56-0297-49b4-a716-f03385219dc0","Type":"ContainerStarted","Data":"e8d113062d7c77a525bc5bb00f97c97d94163b3990bebbe2b8c6f973b964df1f"}
Nov 02 13:49:21.799978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:21.799915    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k" podStartSLOduration=2.143015151 podStartE2EDuration="4.799902696s" totalImagesPullingTime="2.656887545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:21.799612148 +0000 UTC m=+338.377701015" watchObservedRunningTime="2026-11-02 13:49:21.799902696 +0000 UTC m=+338.377991612"
Nov 02 13:49:23.776901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:23.776867    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d" event={"ID":"0c20ef56-0297-49b4-a716-f03385219dc0","Type":"ContainerStarted","Data":"2fdc7e6e5e14d79ef0441154e59939728b3be326abf7dac3e40e272c1f781bd3"}
Nov 02 13:49:23.777417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:23.776996    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:23.821024 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:23.820976    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d" podStartSLOduration=2.251207811 podStartE2EDuration="3.820959278s" totalImagesPullingTime="1.569751467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:23.819162053 +0000 UTC m=+340.397250934" watchObservedRunningTime="2026-11-02 13:49:23.820959278 +0000 UTC m=+340.399048145"
Nov 02 13:49:32.768716 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:32.768690    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:32.987299 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:32.987266    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"]
Nov 02 13:49:32.987541 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:32.987517    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k" podUID="c4601a48-fa26-42af-92b3-3a482313aebb" containerName="limitador" containerID="cri-o://545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0" gracePeriod=30
Nov 02 13:49:33.515701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.515678    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:33.573422 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.573389    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4601a48-fa26-42af-92b3-3a482313aebb-kube-api-access-hhj7m\" (UniqueName: \"kubernetes.io/projected/c4601a48-fa26-42af-92b3-3a482313aebb-kube-api-access-hhj7m\") pod \"c4601a48-fa26-42af-92b3-3a482313aebb\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") "
Nov 02 13:49:33.573645 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.573467    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4601a48-fa26-42af-92b3-3a482313aebb-config-file\" (UniqueName: \"kubernetes.io/configmap/c4601a48-fa26-42af-92b3-3a482313aebb-config-file\") pod \"c4601a48-fa26-42af-92b3-3a482313aebb\" (UID: \"c4601a48-fa26-42af-92b3-3a482313aebb\") "
Nov 02 13:49:33.573875 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.573841    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4601a48-fa26-42af-92b3-3a482313aebb-config-file" pod "c4601a48-fa26-42af-92b3-3a482313aebb" (UID: "c4601a48-fa26-42af-92b3-3a482313aebb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:49:33.575790 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.575769    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4601a48-fa26-42af-92b3-3a482313aebb-kube-api-access-hhj7m" pod "c4601a48-fa26-42af-92b3-3a482313aebb" (UID: "c4601a48-fa26-42af-92b3-3a482313aebb"). InnerVolumeSpecName "kube-api-access-hhj7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:49:33.674393 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.674356    5572 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4601a48-fa26-42af-92b3-3a482313aebb-config-file\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:49:33.674393 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.674389    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhj7m\" (UniqueName: \"kubernetes.io/projected/c4601a48-fa26-42af-92b3-3a482313aebb-kube-api-access-hhj7m\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:49:33.815303 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.815272    5572 generic.go:356] "Generic (PLEG): container finished" podID="c4601a48-fa26-42af-92b3-3a482313aebb" containerID="545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0" exitCode=0
Nov 02 13:49:33.815731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.815314    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k" event={"ID":"c4601a48-fa26-42af-92b3-3a482313aebb","Type":"ContainerDied","Data":"545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0"}
Nov 02 13:49:33.815731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.815321    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"
Nov 02 13:49:33.815731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.815352    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vhb9k" event={"ID":"c4601a48-fa26-42af-92b3-3a482313aebb","Type":"ContainerDied","Data":"406c2513ad5ac005fd2fea619e8eaf70dec5ba6dc9041264f5c26f874355cb87"}
Nov 02 13:49:33.815731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.815373    5572 scope.go:122] "RemoveContainer" containerID="545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0"
Nov 02 13:49:33.823927 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.823906    5572 scope.go:122] "RemoveContainer" containerID="545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0"
Nov 02 13:49:33.824193 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:49:33.824173    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0\": container with ID starting with 545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0 not found: ID does not exist" containerID="545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0"
Nov 02 13:49:33.824275 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.824206    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0"} err="failed to get container status \"545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0\": rpc error: code = NotFound desc = could not find container \"545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0\": container with ID starting with 545d0061898b1cb349b66126e5d48f836adc4041c3fb5c686679a53bd30e0cc0 not found: ID does not exist"
Nov 02 13:49:33.846808 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.846779    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"]
Nov 02 13:49:33.857764 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:33.857743    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vhb9k"]
Nov 02 13:49:35.540611 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:35.540549    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4601a48-fa26-42af-92b3-3a482313aebb" path="/var/lib/kubelet/pods/c4601a48-fa26-42af-92b3-3a482313aebb/volumes"
Nov 02 13:49:39.895791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.895758    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"]
Nov 02 13:49:39.896151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.896076    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c4601a48-fa26-42af-92b3-3a482313aebb" containerName="limitador"
Nov 02 13:49:39.896151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.896087    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4601a48-fa26-42af-92b3-3a482313aebb" containerName="limitador"
Nov 02 13:49:39.896234 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.896155    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4601a48-fa26-42af-92b3-3a482313aebb" containerName="limitador"
Nov 02 13:49:39.900371 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.900354    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:39.905935 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.905912    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 13:49:39.908596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.908573    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xcjxm\""
Nov 02 13:49:39.910017 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.909994    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:49:39.910104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.910033    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:49:39.910822 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.910803    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"]
Nov 02 13:49:39.929516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.929491    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1da4efa5-63bb-45bc-90fc-0ceca7c2e04e-data\") pod \"postgres-7cdbf6dc44-p9hzc\" (UID: \"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:39.929654 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:39.929543    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsg7n\" (UniqueName: \"kubernetes.io/projected/1da4efa5-63bb-45bc-90fc-0ceca7c2e04e-kube-api-access-dsg7n\") pod \"postgres-7cdbf6dc44-p9hzc\" (UID: \"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:40.030831 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:40.030799    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1da4efa5-63bb-45bc-90fc-0ceca7c2e04e-data\") pod \"postgres-7cdbf6dc44-p9hzc\" (UID: \"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:40.030990 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:40.030844    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsg7n\" (UniqueName: \"kubernetes.io/projected/1da4efa5-63bb-45bc-90fc-0ceca7c2e04e-kube-api-access-dsg7n\") pod \"postgres-7cdbf6dc44-p9hzc\" (UID: \"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:40.031201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:40.031166    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1da4efa5-63bb-45bc-90fc-0ceca7c2e04e-data\") pod \"postgres-7cdbf6dc44-p9hzc\" (UID: \"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:40.050450 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:40.050416    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsg7n\" (UniqueName: \"kubernetes.io/projected/1da4efa5-63bb-45bc-90fc-0ceca7c2e04e-kube-api-access-dsg7n\") pod \"postgres-7cdbf6dc44-p9hzc\" (UID: \"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:40.212941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:40.212852    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:40.486635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:40.486542    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"]
Nov 02 13:49:40.489622 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:49:40.489595    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da4efa5_63bb_45bc_90fc_0ceca7c2e04e.slice/crio-a8ecdc4a4d815090b27012c8273093b6de0886499fb3c96e7e4814330f41e418 WatchSource:0}: Error finding container a8ecdc4a4d815090b27012c8273093b6de0886499fb3c96e7e4814330f41e418: Status 404 returned error can't find the container with id a8ecdc4a4d815090b27012c8273093b6de0886499fb3c96e7e4814330f41e418
Nov 02 13:49:40.844180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:40.844145    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc" event={"ID":"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e","Type":"ContainerStarted","Data":"a8ecdc4a4d815090b27012c8273093b6de0886499fb3c96e7e4814330f41e418"}
Nov 02 13:49:44.782889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:44.782848    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-vt89d"
Nov 02 13:49:45.810158 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:45.810138    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 13:49:46.877284 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:46.877245    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc" event={"ID":"1da4efa5-63bb-45bc-90fc-0ceca7c2e04e","Type":"ContainerStarted","Data":"12ea4da63d790f3fc7be143000561f3fa3238418d9156561c7ad3ec12e6574ad"}
Nov 02 13:49:46.877773 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:46.877382    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:49:46.896340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:46.896284    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc" podStartSLOduration=2.580592168 podStartE2EDuration="7.896269039s" totalImagesPullingTime="5.315676871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:46.895438688 +0000 UTC m=+363.473527557" watchObservedRunningTime="2026-11-02 13:49:46.896269039 +0000 UTC m=+363.474357907"
Nov 02 13:49:51.428810 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:49:51.428765    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:49:52.910160 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:49:52.910129    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9hzc"
Nov 02 13:50:10.005131 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.005092    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwq2t"]
Nov 02 13:50:10.025936 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.025909    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.040593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.040475    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8n6b9\""
Nov 02 13:50:10.040593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.040519    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 13:50:10.045290 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.045270    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 13:50:10.045417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.045339    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 13:50:10.046101 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.046080    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwq2t"]
Nov 02 13:50:10.211307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.211274    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6b4d\" (UniqueName: \"kubernetes.io/projected/e96be7fe-4780-463e-ac97-78780f699240-kube-api-access-m6b4d\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.211506 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.211317    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-webhook-cert\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.211506 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.211339    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-metrics-certs\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.312269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.312236    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6b4d\" (UniqueName: \"kubernetes.io/projected/e96be7fe-4780-463e-ac97-78780f699240-kube-api-access-m6b4d\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.312451 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.312292    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-webhook-cert\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.312520 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.312484    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-metrics-certs\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.315028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.315003    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-webhook-cert\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.315125 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.315003    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-metrics-certs\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.325288 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.325262    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6b4d\" (UniqueName: \"kubernetes.io/projected/e96be7fe-4780-463e-ac97-78780f699240-kube-api-access-m6b4d\") pod \"maas-controller-579cf7c6d-gwq2t\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") " pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.339184 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.339162    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:10.622744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.622664    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwq2t"]
Nov 02 13:50:10.626383 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:50:10.626340    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96be7fe_4780_463e_ac97_78780f699240.slice/crio-68d91d529dbe20bf2951719660cc22c91132c51b500f6900bfc20ffc8f043cfc WatchSource:0}: Error finding container 68d91d529dbe20bf2951719660cc22c91132c51b500f6900bfc20ffc8f043cfc: Status 404 returned error can't find the container with id 68d91d529dbe20bf2951719660cc22c91132c51b500f6900bfc20ffc8f043cfc
Nov 02 13:50:10.979654 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:10.979553    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwq2t" event={"ID":"e96be7fe-4780-463e-ac97-78780f699240","Type":"ContainerStarted","Data":"68d91d529dbe20bf2951719660cc22c91132c51b500f6900bfc20ffc8f043cfc"}
Nov 02 13:50:13.762920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.762886    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sk6cs"]
Nov 02 13:50:13.766790 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.766769    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:13.776090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.776048    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sk6cs"]
Nov 02 13:50:13.776421 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.776405    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-h7nlf\""
Nov 02 13:50:13.776542 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.776526    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 13:50:13.838735 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.838708    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgjql\" (UniqueName: \"kubernetes.io/projected/852b251e-7221-460c-a861-0b6e71ac2571-kube-api-access-dgjql\") pod \"kserve-controller-manager-776cc66d47-sk6cs\" (UID: \"852b251e-7221-460c-a861-0b6e71ac2571\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:13.838853 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.838745    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/852b251e-7221-460c-a861-0b6e71ac2571-cert\") pod \"kserve-controller-manager-776cc66d47-sk6cs\" (UID: \"852b251e-7221-460c-a861-0b6e71ac2571\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:13.857828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.857805    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-fj5wt"]
Nov 02 13:50:13.860949 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.860930    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:13.874954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.874931    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fj5wt"]
Nov 02 13:50:13.875883 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.875862    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 13:50:13.876014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.875932    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 13:50:13.876014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.875987    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ns9sc\""
Nov 02 13:50:13.939470 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.939438    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a2d9a3-6280-4635-b146-7fa5b060f049-metrics-certs\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:13.939644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.939484    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjql\" (UniqueName: \"kubernetes.io/projected/852b251e-7221-460c-a861-0b6e71ac2571-kube-api-access-dgjql\") pod \"kserve-controller-manager-776cc66d47-sk6cs\" (UID: \"852b251e-7221-460c-a861-0b6e71ac2571\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:13.939644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.939590    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/852b251e-7221-460c-a861-0b6e71ac2571-cert\") pod \"kserve-controller-manager-776cc66d47-sk6cs\" (UID: \"852b251e-7221-460c-a861-0b6e71ac2571\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:13.939742 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.939698    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69a2d9a3-6280-4635-b146-7fa5b060f049-cert\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:13.939742 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.939729    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpg6\" (UniqueName: \"kubernetes.io/projected/69a2d9a3-6280-4635-b146-7fa5b060f049-kube-api-access-clpg6\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:13.942321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.942299    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/852b251e-7221-460c-a861-0b6e71ac2571-cert\") pod \"kserve-controller-manager-776cc66d47-sk6cs\" (UID: \"852b251e-7221-460c-a861-0b6e71ac2571\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:13.959934 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.959908    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjql\" (UniqueName: \"kubernetes.io/projected/852b251e-7221-460c-a861-0b6e71ac2571-kube-api-access-dgjql\") pod \"kserve-controller-manager-776cc66d47-sk6cs\" (UID: \"852b251e-7221-460c-a861-0b6e71ac2571\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:13.994045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.994001    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwq2t" event={"ID":"e96be7fe-4780-463e-ac97-78780f699240","Type":"ContainerStarted","Data":"ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa"}
Nov 02 13:50:13.994190 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:13.994161    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:14.025028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.024921    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-gwq2t" podStartSLOduration=1.85324883 podStartE2EDuration="5.024908724s" totalImagesPullingTime="3.171659894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:14.024668851 +0000 UTC m=+390.602757720" watchObservedRunningTime="2026-11-02 13:50:14.024908724 +0000 UTC m=+390.602997588"
Nov 02 13:50:14.040185 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.040158    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69a2d9a3-6280-4635-b146-7fa5b060f049-cert\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:14.040307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.040193    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpg6\" (UniqueName: \"kubernetes.io/projected/69a2d9a3-6280-4635-b146-7fa5b060f049-kube-api-access-clpg6\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:14.040307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.040288    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a2d9a3-6280-4635-b146-7fa5b060f049-metrics-certs\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:14.042677 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.042656    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a2d9a3-6280-4635-b146-7fa5b060f049-metrics-certs\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:14.042752 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.042730    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69a2d9a3-6280-4635-b146-7fa5b060f049-cert\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:14.058788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.058765    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpg6\" (UniqueName: \"kubernetes.io/projected/69a2d9a3-6280-4635-b146-7fa5b060f049-kube-api-access-clpg6\") pod \"odh-model-controller-769875db56-fj5wt\" (UID: \"69a2d9a3-6280-4635-b146-7fa5b060f049\") " pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:14.081738 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.081719    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:14.173815 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.173786    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:14.223587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.223545    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"]
Nov 02 13:50:14.227391 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.227295    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.243186 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.243161    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"]
Nov 02 13:50:14.343118 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.343091    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/39308b2f-8194-4827-8f2f-b0be25d372e6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.343271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.343125    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8r74\" (UniqueName: \"kubernetes.io/projected/39308b2f-8194-4827-8f2f-b0be25d372e6-kube-api-access-m8r74\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.343271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.343211    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.343271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.343243    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.343271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.343260    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.343426 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.343391    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.343479 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.343448    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.363977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.363954    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sk6cs"]
Nov 02 13:50:14.367454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:50:14.367429    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852b251e_7221_460c_a861_0b6e71ac2571.slice/crio-a4bfed8f16b8de2bc86e97faf040ba69e69c989540c16271a4aeedd03224438f WatchSource:0}: Error finding container a4bfed8f16b8de2bc86e97faf040ba69e69c989540c16271a4aeedd03224438f: Status 404 returned error can't find the container with id a4bfed8f16b8de2bc86e97faf040ba69e69c989540c16271a4aeedd03224438f
Nov 02 13:50:14.444414 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.444392    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.444534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.444447    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/39308b2f-8194-4827-8f2f-b0be25d372e6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.444534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.444474    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8r74\" (UniqueName: \"kubernetes.io/projected/39308b2f-8194-4827-8f2f-b0be25d372e6-kube-api-access-m8r74\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.444676 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.444572    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.444676 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.444618    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.444676 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.444637    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.444833 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.444678    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.445366 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.445331    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.446749 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.446732    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/39308b2f-8194-4827-8f2f-b0be25d372e6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.446898 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.446876    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.447370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.447351    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.447448 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.447376    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.474171 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.474151    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fj5wt"]
Nov 02 13:50:14.477407 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:50:14.477387    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a2d9a3_6280_4635_b146_7fa5b060f049.slice/crio-be0ead622c22daecdf2af3ed8761df03730c66da0b83103c8999f4c721ceb762 WatchSource:0}: Error finding container be0ead622c22daecdf2af3ed8761df03730c66da0b83103c8999f4c721ceb762: Status 404 returned error can't find the container with id be0ead622c22daecdf2af3ed8761df03730c66da0b83103c8999f4c721ceb762
Nov 02 13:50:14.481278 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.481258    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39308b2f-8194-4827-8f2f-b0be25d372e6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.493377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.493361    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8r74\" (UniqueName: \"kubernetes.io/projected/39308b2f-8194-4827-8f2f-b0be25d372e6-kube-api-access-m8r74\") pod \"istiod-openshift-gateway-7ddd9cfdc8-852sj\" (UID: \"39308b2f-8194-4827-8f2f-b0be25d372e6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.546292 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.546272    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:14.839378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.839348    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"]
Nov 02 13:50:14.842432 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:50:14.842403    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39308b2f_8194_4827_8f2f_b0be25d372e6.slice/crio-46443d392c386ccf4929e1ceb8d4019656a5b3deb3e1aebc2b20a7042cd78d99 WatchSource:0}: Error finding container 46443d392c386ccf4929e1ceb8d4019656a5b3deb3e1aebc2b20a7042cd78d99: Status 404 returned error can't find the container with id 46443d392c386ccf4929e1ceb8d4019656a5b3deb3e1aebc2b20a7042cd78d99
Nov 02 13:50:14.844707 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.844667    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:50:14.844795 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:14.844743    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:50:15.002461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.002425    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj" event={"ID":"39308b2f-8194-4827-8f2f-b0be25d372e6","Type":"ContainerStarted","Data":"742a4a194d622c349c3fde274fade96fe61d21ffbc34a70486a657a609ece8eb"}
Nov 02 13:50:15.002673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.002489    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj" event={"ID":"39308b2f-8194-4827-8f2f-b0be25d372e6","Type":"ContainerStarted","Data":"46443d392c386ccf4929e1ceb8d4019656a5b3deb3e1aebc2b20a7042cd78d99"}
Nov 02 13:50:15.002751 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.002691    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:15.004508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.004487    5572 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-852sj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:50:15.004687 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.004527    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj" podUID="39308b2f-8194-4827-8f2f-b0be25d372e6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:50:15.004875 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.004856    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fj5wt" event={"ID":"69a2d9a3-6280-4635-b146-7fa5b060f049","Type":"ContainerStarted","Data":"be0ead622c22daecdf2af3ed8761df03730c66da0b83103c8999f4c721ceb762"}
Nov 02 13:50:15.006705 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.006684    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs" event={"ID":"852b251e-7221-460c-a861-0b6e71ac2571","Type":"ContainerStarted","Data":"a4bfed8f16b8de2bc86e97faf040ba69e69c989540c16271a4aeedd03224438f"}
Nov 02 13:50:15.091150 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:15.090536    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj" podStartSLOduration=1.090523417 podStartE2EDuration="1.090523417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:15.090090667 +0000 UTC m=+391.668179534" watchObservedRunningTime="2026-11-02 13:50:15.090523417 +0000 UTC m=+391.668612283"
Nov 02 13:50:16.013548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.013510    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-852sj"
Nov 02 13:50:16.087753 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.087063    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"]
Nov 02 13:50:16.087753 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.087268    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh" podUID="2a156c79-3139-41f1-b2fe-cf5ac007d864" containerName="discovery" containerID="cri-o://40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13" gracePeriod=30
Nov 02 13:50:16.363498 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.363469    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:50:16.477980 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.477932    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap\") pod \"2a156c79-3139-41f1-b2fe-cf5ac007d864\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") "
Nov 02 13:50:16.478141 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.478024    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-kube-api-access-7cwdp\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-kube-api-access-7cwdp\") pod \"2a156c79-3139-41f1-b2fe-cf5ac007d864\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") "
Nov 02 13:50:16.478141 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.478074    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a156c79-3139-41f1-b2fe-cf5ac007d864-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a156c79-3139-41f1-b2fe-cf5ac007d864-local-certs\") pod \"2a156c79-3139-41f1-b2fe-cf5ac007d864\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") "
Nov 02 13:50:16.478141 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.478113    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-dns-cert\") pod \"2a156c79-3139-41f1-b2fe-cf5ac007d864\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") "
Nov 02 13:50:16.478351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.478152    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-kubeconfig\") pod \"2a156c79-3139-41f1-b2fe-cf5ac007d864\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") "
Nov 02 13:50:16.478351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.478182    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-cacerts\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-cacerts\") pod \"2a156c79-3139-41f1-b2fe-cf5ac007d864\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") "
Nov 02 13:50:16.478351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.478273    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-token\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-token\") pod \"2a156c79-3139-41f1-b2fe-cf5ac007d864\" (UID: \"2a156c79-3139-41f1-b2fe-cf5ac007d864\") "
Nov 02 13:50:16.479550 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.479497    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap" pod "2a156c79-3139-41f1-b2fe-cf5ac007d864" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:50:16.483356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.483328    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-kubeconfig" pod "2a156c79-3139-41f1-b2fe-cf5ac007d864" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:16.485026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.484962    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a156c79-3139-41f1-b2fe-cf5ac007d864-local-certs" pod "2a156c79-3139-41f1-b2fe-cf5ac007d864" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:50:16.485026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.484971    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-cacerts" pod "2a156c79-3139-41f1-b2fe-cf5ac007d864" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:16.485026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.484987    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-token" pod "2a156c79-3139-41f1-b2fe-cf5ac007d864" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:50:16.485246 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.485184    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-kube-api-access-7cwdp" pod "2a156c79-3139-41f1-b2fe-cf5ac007d864" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864"). InnerVolumeSpecName "kube-api-access-7cwdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:50:16.487055 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.487027    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-dns-cert" pod "2a156c79-3139-41f1-b2fe-cf5ac007d864" (UID: "2a156c79-3139-41f1-b2fe-cf5ac007d864"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:16.580018 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.579981    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:16.580018 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.580017    5572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-ca-configmap\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:16.580283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.580032    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cwdp\" (UniqueName: \"kubernetes.io/projected/2a156c79-3139-41f1-b2fe-cf5ac007d864-kube-api-access-7cwdp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:16.580283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.580050    5572 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a156c79-3139-41f1-b2fe-cf5ac007d864-local-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:16.580283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.580065    5572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-csr-dns-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:16.580283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.580079    5572 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-istio-kubeconfig\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:16.580283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:16.580095    5572 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a156c79-3139-41f1-b2fe-cf5ac007d864-cacerts\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:17.019717 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.019683    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs" event={"ID":"852b251e-7221-460c-a861-0b6e71ac2571","Type":"ContainerStarted","Data":"3cf34cce7f77f88f4eb0c8b2d82b23552b36bd32204d8a74adfe39f90863ab18"}
Nov 02 13:50:17.020111 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.019758    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:17.021744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.021705    5572 generic.go:356] "Generic (PLEG): container finished" podID="2a156c79-3139-41f1-b2fe-cf5ac007d864" containerID="40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13" exitCode=0
Nov 02 13:50:17.021867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.021767    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"
Nov 02 13:50:17.021867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.021790    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh" event={"ID":"2a156c79-3139-41f1-b2fe-cf5ac007d864","Type":"ContainerDied","Data":"40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13"}
Nov 02 13:50:17.021867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.021843    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh" event={"ID":"2a156c79-3139-41f1-b2fe-cf5ac007d864","Type":"ContainerDied","Data":"63cedeb3f41e4e7286a36faf5320b1651188424c8b798b7dc1e20d998d141a27"}
Nov 02 13:50:17.022036 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.021869    5572 scope.go:122] "RemoveContainer" containerID="40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13"
Nov 02 13:50:17.031694 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.031676    5572 scope.go:122] "RemoveContainer" containerID="40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13"
Nov 02 13:50:17.032011 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:50:17.031935    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13\": container with ID starting with 40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13 not found: ID does not exist" containerID="40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13"
Nov 02 13:50:17.032011 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.031971    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13"} err="failed to get container status \"40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13\": rpc error: code = NotFound desc = could not find container \"40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13\": container with ID starting with 40e9990df903d46187f8cd6954b38f4fa4b309a9645801da64cf1b1fb1c87f13 not found: ID does not exist"
Nov 02 13:50:17.045306 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.045266    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs" podStartSLOduration=1.469683467 podStartE2EDuration="4.045251477s" totalImagesPullingTime="2.57556801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:17.044251516 +0000 UTC m=+393.622340395" watchObservedRunningTime="2026-11-02 13:50:17.045251477 +0000 UTC m=+393.623340348"
Nov 02 13:50:17.059478 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.059458    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"]
Nov 02 13:50:17.064478 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.064459    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lf6qh"]
Nov 02 13:50:17.541242 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:17.541219    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a156c79-3139-41f1-b2fe-cf5ac007d864" path="/var/lib/kubelet/pods/2a156c79-3139-41f1-b2fe-cf5ac007d864/volumes"
Nov 02 13:50:18.029608 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:18.029534    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fj5wt" event={"ID":"69a2d9a3-6280-4635-b146-7fa5b060f049","Type":"ContainerStarted","Data":"ae07000f9670cd5521475b9c9a708dc2fb17b711455a3c074ea8d972d70e5861"}
Nov 02 13:50:18.030014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:18.029653    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:18.049883 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:18.049835    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-fj5wt" podStartSLOduration=2.086020705 podStartE2EDuration="5.049822257s" totalImagesPullingTime="2.963801552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:18.049075019 +0000 UTC m=+394.627163887" watchObservedRunningTime="2026-11-02 13:50:18.049822257 +0000 UTC m=+394.627911163"
Nov 02 13:50:21.428942 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:50:21.428902    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:50:23.876378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.876349    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"]
Nov 02 13:50:23.876739 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.876711    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2a156c79-3139-41f1-b2fe-cf5ac007d864" containerName="discovery"
Nov 02 13:50:23.876739 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.876722    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a156c79-3139-41f1-b2fe-cf5ac007d864" containerName="discovery"
Nov 02 13:50:23.876820 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.876807    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a156c79-3139-41f1-b2fe-cf5ac007d864" containerName="discovery"
Nov 02 13:50:23.879716 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.879696    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:23.886819 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.886785    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 13:50:23.886922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.886839    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-7l2nl\""
Nov 02 13:50:23.886922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.886874    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 13:50:23.886922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.886874    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 13:50:23.886922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.886880    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:50:23.886922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.886915    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 13:50:23.895972 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.895953    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"]
Nov 02 13:50:23.946148 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.946128    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a84319-eb06-4af8-9383-410a49087ea0-metrics-certs\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:23.946271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.946203    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83a84319-eb06-4af8-9383-410a49087ea0-cert\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:23.946443 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.946413    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/83a84319-eb06-4af8-9383-410a49087ea0-manager-config\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:23.946528 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:23.946509    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9668\" (UniqueName: \"kubernetes.io/projected/83a84319-eb06-4af8-9383-410a49087ea0-kube-api-access-z9668\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.047351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.047324    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83a84319-eb06-4af8-9383-410a49087ea0-cert\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.047450 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.047372    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/83a84319-eb06-4af8-9383-410a49087ea0-manager-config\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.047450 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.047395    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9668\" (UniqueName: \"kubernetes.io/projected/83a84319-eb06-4af8-9383-410a49087ea0-kube-api-access-z9668\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.047613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.047594    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a84319-eb06-4af8-9383-410a49087ea0-metrics-certs\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.048098 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.048079    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/83a84319-eb06-4af8-9383-410a49087ea0-manager-config\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.049995 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.049977    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83a84319-eb06-4af8-9383-410a49087ea0-cert\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.050104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.050085    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a84319-eb06-4af8-9383-410a49087ea0-metrics-certs\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.060719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.060696    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9668\" (UniqueName: \"kubernetes.io/projected/83a84319-eb06-4af8-9383-410a49087ea0-kube-api-access-z9668\") pod \"jobset-controller-manager-784f4d6794-g7pn8\" (UID: \"83a84319-eb06-4af8-9383-410a49087ea0\") " pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.192608 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.192529    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:24.501339 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:24.501255    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"]
Nov 02 13:50:24.505688 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:50:24.505655    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a84319_eb06_4af8_9383_410a49087ea0.slice/crio-ca3e2c08f184e43dd5d44ce033b1d647b724b8c7c2617d125ad7b8f0392f10a0 WatchSource:0}: Error finding container ca3e2c08f184e43dd5d44ce033b1d647b724b8c7c2617d125ad7b8f0392f10a0: Status 404 returned error can't find the container with id ca3e2c08f184e43dd5d44ce033b1d647b724b8c7c2617d125ad7b8f0392f10a0
Nov 02 13:50:25.009815 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.009791    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:25.058406 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.058372    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8" event={"ID":"83a84319-eb06-4af8-9383-410a49087ea0","Type":"ContainerStarted","Data":"ca3e2c08f184e43dd5d44ce033b1d647b724b8c7c2617d125ad7b8f0392f10a0"}
Nov 02 13:50:25.144411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.144378    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwq2t"]
Nov 02 13:50:25.144617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.144596    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-gwq2t" podUID="e96be7fe-4780-463e-ac97-78780f699240" containerName="manager" containerID="cri-o://ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa" gracePeriod=30
Nov 02 13:50:25.376432 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.376409    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:25.460942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.460912    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96be7fe-4780-463e-ac97-78780f699240-kube-api-access-m6b4d\" (UniqueName: \"kubernetes.io/projected/e96be7fe-4780-463e-ac97-78780f699240-kube-api-access-m6b4d\") pod \"e96be7fe-4780-463e-ac97-78780f699240\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") "
Nov 02 13:50:25.461081 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.461010    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-webhook-cert\") pod \"e96be7fe-4780-463e-ac97-78780f699240\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") "
Nov 02 13:50:25.461081 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.461060    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-metrics-certs\") pod \"e96be7fe-4780-463e-ac97-78780f699240\" (UID: \"e96be7fe-4780-463e-ac97-78780f699240\") "
Nov 02 13:50:25.463256 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.463217    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-webhook-cert" pod "e96be7fe-4780-463e-ac97-78780f699240" (UID: "e96be7fe-4780-463e-ac97-78780f699240"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:25.463385 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.463295    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-metrics-certs" pod "e96be7fe-4780-463e-ac97-78780f699240" (UID: "e96be7fe-4780-463e-ac97-78780f699240"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:25.463385 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.463338    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96be7fe-4780-463e-ac97-78780f699240-kube-api-access-m6b4d" pod "e96be7fe-4780-463e-ac97-78780f699240" (UID: "e96be7fe-4780-463e-ac97-78780f699240"). InnerVolumeSpecName "kube-api-access-m6b4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:50:25.561926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.561869    5572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-webhook-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:25.561926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.561890    5572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e96be7fe-4780-463e-ac97-78780f699240-metrics-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:25.561926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:25.561903    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6b4d\" (UniqueName: \"kubernetes.io/projected/e96be7fe-4780-463e-ac97-78780f699240-kube-api-access-m6b4d\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:50:26.063784 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.063748    5572 generic.go:356] "Generic (PLEG): container finished" podID="e96be7fe-4780-463e-ac97-78780f699240" containerID="ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa" exitCode=0
Nov 02 13:50:26.064161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.063794    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwq2t"
Nov 02 13:50:26.064161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.063829    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwq2t" event={"ID":"e96be7fe-4780-463e-ac97-78780f699240","Type":"ContainerDied","Data":"ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa"}
Nov 02 13:50:26.064161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.063885    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwq2t" event={"ID":"e96be7fe-4780-463e-ac97-78780f699240","Type":"ContainerDied","Data":"68d91d529dbe20bf2951719660cc22c91132c51b500f6900bfc20ffc8f043cfc"}
Nov 02 13:50:26.064161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.063908    5572 scope.go:122] "RemoveContainer" containerID="ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa"
Nov 02 13:50:26.071958 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.071945    5572 scope.go:122] "RemoveContainer" containerID="ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa"
Nov 02 13:50:26.072231 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:50:26.072203    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa\": container with ID starting with ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa not found: ID does not exist" containerID="ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa"
Nov 02 13:50:26.072325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.072238    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa"} err="failed to get container status \"ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa\": rpc error: code = NotFound desc = could not find container \"ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa\": container with ID starting with ff45ee255203d289ad5e0263778800b79e34d9ed23c4f69d6fcd4da3df8985aa not found: ID does not exist"
Nov 02 13:50:26.083762 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.083741    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwq2t"]
Nov 02 13:50:26.087486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:26.087467    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwq2t"]
Nov 02 13:50:27.541172 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:27.541138    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96be7fe-4780-463e-ac97-78780f699240" path="/var/lib/kubelet/pods/e96be7fe-4780-463e-ac97-78780f699240/volumes"
Nov 02 13:50:29.036646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:29.036616    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-fj5wt"
Nov 02 13:50:29.081193 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:29.081167    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8" event={"ID":"83a84319-eb06-4af8-9383-410a49087ea0","Type":"ContainerStarted","Data":"90a79bd633e6bc73b2c4543de6015555b7d73eac53099c276fbdd4b8d9c5ebe4"}
Nov 02 13:50:29.081319 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:29.081306    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:29.157275 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:29.157205    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8" podStartSLOduration=2.495622081 podStartE2EDuration="6.15719003s" totalImagesPullingTime="3.661567949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:29.156244695 +0000 UTC m=+405.734333564" watchObservedRunningTime="2026-11-02 13:50:29.15719003 +0000 UTC m=+405.735278896"
Nov 02 13:50:40.090293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:40.090266    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-784f4d6794-g7pn8"
Nov 02 13:50:48.035947 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:50:48.035919    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-sk6cs"
Nov 02 13:50:51.430376 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:50:51.430334    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:51:21.428312 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:51:21.428274    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:51:51.429008 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:51:51.428935    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:52:21.428647 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:52:21.428610    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:52:25.577760 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.577728    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"]
Nov 02 13:52:25.578168 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.578141    5572 cpu_manager.go:414] "containerMap: removing container" podUID="e96be7fe-4780-463e-ac97-78780f699240" containerName="manager"
Nov 02 13:52:25.578168 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.578154    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96be7fe-4780-463e-ac97-78780f699240" containerName="manager"
Nov 02 13:52:25.578247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.578216    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96be7fe-4780-463e-ac97-78780f699240" containerName="manager"
Nov 02 13:52:25.581009 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.580989    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.586040 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.586017    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lwlfp\""
Nov 02 13:52:25.586162 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.586089    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 13:52:25.586162 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.586096    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 13:52:25.586162 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.586122    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 13:52:25.591439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.591408    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"]
Nov 02 13:52:25.663817 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.663793    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.663994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.663825    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.663994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.663844    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.663994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.663925    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7fj\" (UniqueName: \"kubernetes.io/projected/87666c0f-6310-470a-8242-b4345b50acbf-kube-api-access-dn7fj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.664137 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.664007    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87666c0f-6310-470a-8242-b4345b50acbf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.664137 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.664096    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.764879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.764851    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.764879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.764883    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.765097 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.764914    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7fj\" (UniqueName: \"kubernetes.io/projected/87666c0f-6310-470a-8242-b4345b50acbf-kube-api-access-dn7fj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.765097 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.764952    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87666c0f-6310-470a-8242-b4345b50acbf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.765097 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.764989    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.765097 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.765028    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.765304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.765280    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.765366 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.765336    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.765650 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.765627    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.767336 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.767316    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87666c0f-6310-470a-8242-b4345b50acbf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.767638 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.767621    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87666c0f-6310-470a-8242-b4345b50acbf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.777156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.777132    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7fj\" (UniqueName: \"kubernetes.io/projected/87666c0f-6310-470a-8242-b4345b50acbf-kube-api-access-dn7fj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2\" (UID: \"87666c0f-6310-470a-8242-b4345b50acbf\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:25.899300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:25.899239    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:26.174100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:26.173901    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"]
Nov 02 13:52:26.176721 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:52:26.176691    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87666c0f_6310_470a_8242_b4345b50acbf.slice/crio-75f06a2fca94b97b731a0e82c612ba590ae7c73d9d5ae8c9be0680d80bcb9dbc WatchSource:0}: Error finding container 75f06a2fca94b97b731a0e82c612ba590ae7c73d9d5ae8c9be0680d80bcb9dbc: Status 404 returned error can't find the container with id 75f06a2fca94b97b731a0e82c612ba590ae7c73d9d5ae8c9be0680d80bcb9dbc
Nov 02 13:52:26.574370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:26.574329    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" event={"ID":"87666c0f-6310-470a-8242-b4345b50acbf","Type":"ContainerStarted","Data":"75f06a2fca94b97b731a0e82c612ba590ae7c73d9d5ae8c9be0680d80bcb9dbc"}
Nov 02 13:52:27.580149 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:27.580110    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" event={"ID":"87666c0f-6310-470a-8242-b4345b50acbf","Type":"ContainerStarted","Data":"4e3a8fc940136d528e5886a78b5b47bacc34b75a36dc68e51a7a38c8971cad0e"}
Nov 02 13:52:27.600843 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:27.600795    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podStartSLOduration=1.386693409 podStartE2EDuration="2.6007826s" totalImagesPullingTime="1.214089191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:27.599822742 +0000 UTC m=+524.177911621" watchObservedRunningTime="2026-11-02 13:52:27.6007826 +0000 UTC m=+524.178871466"
Nov 02 13:52:27.900158 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:27.900123    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:28.357435 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.357399    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"]
Nov 02 13:52:28.360922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.360902    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.371459 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.371434    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 13:52:28.374628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.374608    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"]
Nov 02 13:52:28.487402 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.487355    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.487819 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.487778    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.487943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.487841    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.488009 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.487982    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zh9\" (UniqueName: \"kubernetes.io/projected/6b462672-55fe-42b8-b39f-52b0de99145a-kube-api-access-28zh9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.488064 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.488042    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.488107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.488064    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b462672-55fe-42b8-b39f-52b0de99145a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589010    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589070    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zh9\" (UniqueName: \"kubernetes.io/projected/6b462672-55fe-42b8-b39f-52b0de99145a-kube-api-access-28zh9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589119    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589147    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b462672-55fe-42b8-b39f-52b0de99145a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589200    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589226    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589442    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589475    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.589965 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.589767    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.592233 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.592201    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b462672-55fe-42b8-b39f-52b0de99145a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.592365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.592288    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b462672-55fe-42b8-b39f-52b0de99145a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.605052 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.605025    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zh9\" (UniqueName: \"kubernetes.io/projected/6b462672-55fe-42b8-b39f-52b0de99145a-kube-api-access-28zh9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q595n\" (UID: \"6b462672-55fe-42b8-b39f-52b0de99145a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.674361 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.674279    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:28.947972 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:28.947885    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"]
Nov 02 13:52:28.950819 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:52:28.950782    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b462672_55fe_42b8_b39f_52b0de99145a.slice/crio-e2c595e06215d3f4b4a7d8b47568c450b91f42defb8d85e51aba03ce04ffae54 WatchSource:0}: Error finding container e2c595e06215d3f4b4a7d8b47568c450b91f42defb8d85e51aba03ce04ffae54: Status 404 returned error can't find the container with id e2c595e06215d3f4b4a7d8b47568c450b91f42defb8d85e51aba03ce04ffae54
Nov 02 13:52:29.590359 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:29.590322    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n" event={"ID":"6b462672-55fe-42b8-b39f-52b0de99145a","Type":"ContainerStarted","Data":"e4b1cd7eb30afee0e6ad37045773c0d6119e70f955996049c6f703b8dd648e55"}
Nov 02 13:52:29.590775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:29.590375    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n" event={"ID":"6b462672-55fe-42b8-b39f-52b0de99145a","Type":"ContainerStarted","Data":"e2c595e06215d3f4b4a7d8b47568c450b91f42defb8d85e51aba03ce04ffae54"}
Nov 02 13:52:29.611411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:29.611045    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n" podStartSLOduration=1.282323896 podStartE2EDuration="1.6110313s" totalImagesPullingTime="328.707404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:29.610150283 +0000 UTC m=+526.188239161" watchObservedRunningTime="2026-11-02 13:52:29.6110313 +0000 UTC m=+526.189120164"
Nov 02 13:52:29.674721 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:29.674693    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:35.899661 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:35.899630    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:35.912603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:35.912572    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:36.629491 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:36.629461    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:52:38.675283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:38.675258    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:38.687630 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:38.687605    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:39.641157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:39.641127    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n"
Nov 02 13:52:41.497189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.497160    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"]
Nov 02 13:52:41.500928 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.500904    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:41.511434 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.511409    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 13:52:41.513903 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.513883    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"]
Nov 02 13:52:41.514807 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.514791    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 13:52:41.514900 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.514825    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-tmcwp\""
Nov 02 13:52:41.608419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.608389    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghwvf\" (UniqueName: \"kubernetes.io/projected/8c05a58d-1880-4b4e-9ee8-7030d42e5688-kube-api-access-ghwvf\") pod \"ai-gateway-controller-6cf7bcbfb6-rpvrg\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:41.608620 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.608444    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c05a58d-1880-4b4e-9ee8-7030d42e5688-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-rpvrg\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:41.709511 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.709486    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c05a58d-1880-4b4e-9ee8-7030d42e5688-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-rpvrg\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:41.709728 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.709553    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwvf\" (UniqueName: \"kubernetes.io/projected/8c05a58d-1880-4b4e-9ee8-7030d42e5688-kube-api-access-ghwvf\") pod \"ai-gateway-controller-6cf7bcbfb6-rpvrg\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:41.712085 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.712067    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c05a58d-1880-4b4e-9ee8-7030d42e5688-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-rpvrg\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:41.723665 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.723634    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwvf\" (UniqueName: \"kubernetes.io/projected/8c05a58d-1880-4b4e-9ee8-7030d42e5688-kube-api-access-ghwvf\") pod \"ai-gateway-controller-6cf7bcbfb6-rpvrg\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:41.813872 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:41.813843    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:42.090625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:42.090528    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"]
Nov 02 13:52:42.094106 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:52:42.094077    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c05a58d_1880_4b4e_9ee8_7030d42e5688.slice/crio-f105411265259fa86d09dc05bcee656a7c8e8ccc854121f52c13feb3a6b8f266 WatchSource:0}: Error finding container f105411265259fa86d09dc05bcee656a7c8e8ccc854121f52c13feb3a6b8f266: Status 404 returned error can't find the container with id f105411265259fa86d09dc05bcee656a7c8e8ccc854121f52c13feb3a6b8f266
Nov 02 13:52:42.645675 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:42.645640    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg" event={"ID":"8c05a58d-1880-4b4e-9ee8-7030d42e5688","Type":"ContainerStarted","Data":"f105411265259fa86d09dc05bcee656a7c8e8ccc854121f52c13feb3a6b8f266"}
Nov 02 13:52:46.426251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:46.426230    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 13:52:46.663748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:46.663714    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg" event={"ID":"8c05a58d-1880-4b4e-9ee8-7030d42e5688","Type":"ContainerStarted","Data":"eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9"}
Nov 02 13:52:46.663937 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:46.663846    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:46.695642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:46.695525    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg" podStartSLOduration=1.368836274 podStartE2EDuration="5.695511253s" totalImagesPullingTime="4.326674979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:46.694888156 +0000 UTC m=+543.272977045" watchObservedRunningTime="2026-11-02 13:52:46.695511253 +0000 UTC m=+543.273600120"
Nov 02 13:52:47.327028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.327002    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"]
Nov 02 13:52:47.330665 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.330638    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.344428 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.344400    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ql6jd\""
Nov 02 13:52:47.362897 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.362865    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363050 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.362908    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363050 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.362931    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljh8\" (UniqueName: \"kubernetes.io/projected/f7165380-1253-4546-b0bf-f80b2df80754-kube-api-access-kljh8\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363050 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.363015    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7165380-1253-4546-b0bf-f80b2df80754-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363050 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.363043    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7165380-1253-4546-b0bf-f80b2df80754-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.363060    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7165380-1253-4546-b0bf-f80b2df80754-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.363149    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.363175    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.363312 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.363266    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.365443 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.365416    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"]
Nov 02 13:52:47.461626 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.461595    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"]
Nov 02 13:52:47.464366 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464339    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464383    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464400    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464522    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kljh8\" (UniqueName: \"kubernetes.io/projected/f7165380-1253-4546-b0bf-f80b2df80754-kube-api-access-kljh8\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464592    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7165380-1253-4546-b0bf-f80b2df80754-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464620    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7165380-1253-4546-b0bf-f80b2df80754-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464738 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464649    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7165380-1253-4546-b0bf-f80b2df80754-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464738 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464704    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464738    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.464834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.464787    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.465159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.465140    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.465270 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.465157    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.465331 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.465189    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.465385 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.465352    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7165380-1253-4546-b0bf-f80b2df80754-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.466966 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.466934    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.467079 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.467051    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7165380-1253-4546-b0bf-f80b2df80754-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.467460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.467444    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7165380-1253-4546-b0bf-f80b2df80754-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.471879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.471840    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m4twz\""
Nov 02 13:52:47.475074 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.473403    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gws8m"]
Nov 02 13:52:47.477576 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.477538    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"]
Nov 02 13:52:47.477706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.477686    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.487666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.487579    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 13:52:47.490096 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.490073    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7165380-1253-4546-b0bf-f80b2df80754-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.490468 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.490447    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljh8\" (UniqueName: \"kubernetes.io/projected/f7165380-1253-4546-b0bf-f80b2df80754-kube-api-access-kljh8\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-bnw6z\" (UID: \"f7165380-1253-4546-b0bf-f80b2df80754\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.497211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.497172    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gws8m"]
Nov 02 13:52:47.565377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.565346    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb68a58-70dd-4487-8da8-542ad36afcea-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gws8m\" (UID: \"abb68a58-70dd-4487-8da8-542ad36afcea\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.565621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.565388    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kbtw9\" (UID: \"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.565621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.565527    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zd4\" (UniqueName: \"kubernetes.io/projected/ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7-kube-api-access-l8zd4\") pod \"payload-pre-processing-7b98bb678c-kbtw9\" (UID: \"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.565621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.565590    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrjxp\" (UniqueName: \"kubernetes.io/projected/abb68a58-70dd-4487-8da8-542ad36afcea-kube-api-access-hrjxp\") pod \"payload-processing-55fbbbf9dc-gws8m\" (UID: \"abb68a58-70dd-4487-8da8-542ad36afcea\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.645797 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.645719    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:47.666432 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.666409    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb68a58-70dd-4487-8da8-542ad36afcea-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gws8m\" (UID: \"abb68a58-70dd-4487-8da8-542ad36afcea\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.666549 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.666446    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kbtw9\" (UID: \"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.666549 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.666501    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zd4\" (UniqueName: \"kubernetes.io/projected/ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7-kube-api-access-l8zd4\") pod \"payload-pre-processing-7b98bb678c-kbtw9\" (UID: \"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.666549 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.666529    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrjxp\" (UniqueName: \"kubernetes.io/projected/abb68a58-70dd-4487-8da8-542ad36afcea-kube-api-access-hrjxp\") pod \"payload-processing-55fbbbf9dc-gws8m\" (UID: \"abb68a58-70dd-4487-8da8-542ad36afcea\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.667479 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.667189    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb68a58-70dd-4487-8da8-542ad36afcea-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gws8m\" (UID: \"abb68a58-70dd-4487-8da8-542ad36afcea\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.667479 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.667243    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kbtw9\" (UID: \"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.679437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.679415    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrjxp\" (UniqueName: \"kubernetes.io/projected/abb68a58-70dd-4487-8da8-542ad36afcea-kube-api-access-hrjxp\") pod \"payload-processing-55fbbbf9dc-gws8m\" (UID: \"abb68a58-70dd-4487-8da8-542ad36afcea\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.679596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.679547    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zd4\" (UniqueName: \"kubernetes.io/projected/ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7-kube-api-access-l8zd4\") pod \"payload-pre-processing-7b98bb678c-kbtw9\" (UID: \"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.812930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.812902    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:47.819195 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.819172    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:47.933214 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:47.933083    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"]
Nov 02 13:52:47.935269 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:52:47.935237    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7165380_1253_4546_b0bf_f80b2df80754.slice/crio-efc54cb592d1636bff2073fc9c11038fea305de32bcface98f40662947dd31f3 WatchSource:0}: Error finding container efc54cb592d1636bff2073fc9c11038fea305de32bcface98f40662947dd31f3: Status 404 returned error can't find the container with id efc54cb592d1636bff2073fc9c11038fea305de32bcface98f40662947dd31f3
Nov 02 13:52:48.091383 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:48.091350    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"]
Nov 02 13:52:48.094516 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:52:48.094490    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2cfc9f_85e6_48c9_b3a9_3f6c168b92b7.slice/crio-2770af7a629b22f8285f2e52f71ce7b4f0fcf55eb362a9dd5fedd74cf4fe66fd WatchSource:0}: Error finding container 2770af7a629b22f8285f2e52f71ce7b4f0fcf55eb362a9dd5fedd74cf4fe66fd: Status 404 returned error can't find the container with id 2770af7a629b22f8285f2e52f71ce7b4f0fcf55eb362a9dd5fedd74cf4fe66fd
Nov 02 13:52:48.098804 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:48.098777    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gws8m"]
Nov 02 13:52:48.101412 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:52:48.101390    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb68a58_70dd_4487_8da8_542ad36afcea.slice/crio-271638a5522e6e24c030b14edd1ac2c6af502c97be297fb2720e9713a70230cb WatchSource:0}: Error finding container 271638a5522e6e24c030b14edd1ac2c6af502c97be297fb2720e9713a70230cb: Status 404 returned error can't find the container with id 271638a5522e6e24c030b14edd1ac2c6af502c97be297fb2720e9713a70230cb
Nov 02 13:52:48.680601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:48.680538    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" event={"ID":"f7165380-1253-4546-b0bf-f80b2df80754","Type":"ContainerStarted","Data":"efc54cb592d1636bff2073fc9c11038fea305de32bcface98f40662947dd31f3"}
Nov 02 13:52:48.682178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:48.682148    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9" event={"ID":"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7","Type":"ContainerStarted","Data":"2770af7a629b22f8285f2e52f71ce7b4f0fcf55eb362a9dd5fedd74cf4fe66fd"}
Nov 02 13:52:48.684616 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:48.684591    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m" event={"ID":"abb68a58-70dd-4487-8da8-542ad36afcea","Type":"ContainerStarted","Data":"271638a5522e6e24c030b14edd1ac2c6af502c97be297fb2720e9713a70230cb"}
Nov 02 13:52:50.696613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:50.696511    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m" event={"ID":"abb68a58-70dd-4487-8da8-542ad36afcea","Type":"ContainerStarted","Data":"f6ff51810e334fb791d3c00894228499641f7b87c9ef41c41f9c8ec45edfa60b"}
Nov 02 13:52:50.697052 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:50.696700    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:52:50.698244 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:50.698220    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9" event={"ID":"ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7","Type":"ContainerStarted","Data":"bc108622f157e4af1b5d4df3e9033443e8097e9676e247d6250408612ebb504c"}
Nov 02 13:52:50.698401 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:50.698382    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:52:50.715123 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:50.715075    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m" podStartSLOduration=1.380258658 podStartE2EDuration="3.71506329s" totalImagesPullingTime="2.334804632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:50.714873362 +0000 UTC m=+547.292962233" watchObservedRunningTime="2026-11-02 13:52:50.71506329 +0000 UTC m=+547.293152156"
Nov 02 13:52:50.733996 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:50.733947    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9" podStartSLOduration=1.333730783 podStartE2EDuration="3.733930806s" totalImagesPullingTime="2.400200023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:50.732546692 +0000 UTC m=+547.310635560" watchObservedRunningTime="2026-11-02 13:52:50.733930806 +0000 UTC m=+547.312019698"
Nov 02 13:52:51.434856 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:52:51.434735    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:52:51.624732 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:51.624701    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:52:51.624862 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:51.624778    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:52:51.624862 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:51.624845    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:52:51.704761 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:51.704683    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" event={"ID":"f7165380-1253-4546-b0bf-f80b2df80754","Type":"ContainerStarted","Data":"92270a09ca58db0b444c74f49f70fbf142b07b03a0bd107a5f3ee95a0f4a0aa5"}
Nov 02 13:52:51.728855 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:51.728797    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podStartSLOduration=1.042196446 podStartE2EDuration="4.7287815s" totalImagesPullingTime="3.686585054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:51.727684339 +0000 UTC m=+548.305773208" watchObservedRunningTime="2026-11-02 13:52:51.7287815 +0000 UTC m=+548.306870438"
Nov 02 13:52:52.646933 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:52.646904    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:52.648398 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:52.648370    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 13:52:52.648523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:52.648429    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 13:52:53.647061 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:53.647026    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 13:52:53.647455 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:53.647082    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 13:52:54.646161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:54.646126    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 13:52:54.646340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:54.646177    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 13:52:55.646901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:55.646869    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 13:52:55.647345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:55.646922    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 13:52:56.646500 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:56.646468    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 13:52:56.646698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:56.646521    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 13:52:57.646863 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:57.646829    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 13:52:57.647269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:57.646879    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 13:52:57.671669 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:57.671646    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:52:58.646871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:58.646837    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 13:52:58.647256 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:58.646892    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 13:52:59.801377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:59.801346    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:59.801890 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:59.801768    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:52:59.802433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:52:59.802413    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z"
Nov 02 13:53:01.707469 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:01.707440    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9"
Nov 02 13:53:01.707876 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:01.707515    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gws8m"
Nov 02 13:53:02.339217 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.339181    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"]
Nov 02 13:53:02.345523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.345500    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.359348 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.359322    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"]
Nov 02 13:53:02.373829 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.373802    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"]
Nov 02 13:53:02.378707 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.378473    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.391301 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.391276    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"]
Nov 02 13:53:02.408282 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408254    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408300    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408380    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2cz\" (UniqueName: \"kubernetes.io/projected/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-kube-api-access-gg2cz\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408421    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408440    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408552    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408618    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408635    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.408720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.408690    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.509959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.509919    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.509959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.509967    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510190 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.509992    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2035df-6646-4894-946b-6b38c5d60f77-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510242 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510177    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510241    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510360 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510302    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2035df-6646-4894-946b-6b38c5d60f77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510360 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510335    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510360 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510345    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510388    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510448    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2cz\" (UniqueName: \"kubernetes.io/projected/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-kube-api-access-gg2cz\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510490    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510685 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510519    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510685 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510602    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2035df-6646-4894-946b-6b38c5d60f77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510685 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510662    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510804 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510701    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510804 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510727    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510804 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510756    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.510804 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510790    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzn8s\" (UniqueName: \"kubernetes.io/projected/fc2035df-6646-4894-946b-6b38c5d60f77-kube-api-access-zzn8s\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.510972 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.510828    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.511031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.511013    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.511102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.511073    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.511180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.511161    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.511248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.511163    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.513101 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.513086    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.513327 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.513305    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.523004 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.522979    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.523109 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.523074    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2cz\" (UniqueName: \"kubernetes.io/projected/b508b5b5-3d14-4bc5-8462-a45ec55b2b49-kube-api-access-gg2cz\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-drgtc\" (UID: \"b508b5b5-3d14-4bc5-8462-a45ec55b2b49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.612248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612160    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612195    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2035df-6646-4894-946b-6b38c5d60f77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612215    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612399    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612446    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2035df-6646-4894-946b-6b38c5d60f77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612480    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzn8s\" (UniqueName: \"kubernetes.io/projected/fc2035df-6646-4894-946b-6b38c5d60f77-kube-api-access-zzn8s\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612526    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612547    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2035df-6646-4894-946b-6b38c5d60f77-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612590    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612599    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612653    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.612788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612745    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.613188 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.612867    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.613188 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.613128    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2035df-6646-4894-946b-6b38c5d60f77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.614796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.614769    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2035df-6646-4894-946b-6b38c5d60f77-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.614883 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.614855    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2035df-6646-4894-946b-6b38c5d60f77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.625140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.625120    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2035df-6646-4894-946b-6b38c5d60f77-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.625410 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.625392    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzn8s\" (UniqueName: \"kubernetes.io/projected/fc2035df-6646-4894-946b-6b38c5d60f77-kube-api-access-zzn8s\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-l7sbn\" (UID: \"fc2035df-6646-4894-946b-6b38c5d60f77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.659384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.659365    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:02.693734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.693710    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:02.940048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.940019    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"]
Nov 02 13:53:02.942363 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:02.942339    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb508b5b5_3d14_4bc5_8462_a45ec55b2b49.slice/crio-aacf4f52c0623ab83cf87ab6a974705cef9cb86153d0b19d0ddada39a209b5ac WatchSource:0}: Error finding container aacf4f52c0623ab83cf87ab6a974705cef9cb86153d0b19d0ddada39a209b5ac: Status 404 returned error can't find the container with id aacf4f52c0623ab83cf87ab6a974705cef9cb86153d0b19d0ddada39a209b5ac
Nov 02 13:53:02.945399 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.945368    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:02.945494 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.945442    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:02.945494 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.945487    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:02.988121 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.988099    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"]
Nov 02 13:53:02.989275 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:02.989252    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2035df_6646_4894_946b_6b38c5d60f77.slice/crio-7b246919c545e508636d3c24783e1fc38b23f827572c1d25ebb1305e0a14b77f WatchSource:0}: Error finding container 7b246919c545e508636d3c24783e1fc38b23f827572c1d25ebb1305e0a14b77f: Status 404 returned error can't find the container with id 7b246919c545e508636d3c24783e1fc38b23f827572c1d25ebb1305e0a14b77f
Nov 02 13:53:02.992323 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.992294    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:02.992422 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.992365    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:02.992422 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:02.992407    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:03.761597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:03.761529    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" event={"ID":"fc2035df-6646-4894-946b-6b38c5d60f77","Type":"ContainerStarted","Data":"5f220b7aec25447c6360ff9945bc174994b992fc43c5d8adb7ab47a225b7dcf3"}
Nov 02 13:53:03.762005 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:03.761984    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" event={"ID":"fc2035df-6646-4894-946b-6b38c5d60f77","Type":"ContainerStarted","Data":"7b246919c545e508636d3c24783e1fc38b23f827572c1d25ebb1305e0a14b77f"}
Nov 02 13:53:03.763127 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:03.763107    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" event={"ID":"b508b5b5-3d14-4bc5-8462-a45ec55b2b49","Type":"ContainerStarted","Data":"703fd27e6ffa916bb242e6a858dd017d87d0b747ea0778cba5bcabd0f53701e5"}
Nov 02 13:53:03.763170 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:03.763148    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" event={"ID":"b508b5b5-3d14-4bc5-8462-a45ec55b2b49","Type":"ContainerStarted","Data":"aacf4f52c0623ab83cf87ab6a974705cef9cb86153d0b19d0ddada39a209b5ac"}
Nov 02 13:53:03.831606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:03.826202    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podStartSLOduration=1.826165213 podStartE2EDuration="1.826165213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:03.825913328 +0000 UTC m=+560.404002175" watchObservedRunningTime="2026-11-02 13:53:03.826165213 +0000 UTC m=+560.404254081"
Nov 02 13:53:03.853379 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:03.853326    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podStartSLOduration=1.853311122 podStartE2EDuration="1.853311122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:03.852284617 +0000 UTC m=+560.430373484" watchObservedRunningTime="2026-11-02 13:53:03.853311122 +0000 UTC m=+560.431400044"
Nov 02 13:53:04.660484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:04.660408    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:04.661893 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:04.661873    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 13:53:04.661961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:04.661931    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 13:53:04.699583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:04.697934    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:04.699583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:04.698041    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 13:53:04.699583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:04.698079    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 13:53:05.659641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:05.659613    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 13:53:05.659875 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:05.659853    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 13:53:05.696586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:05.695034    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 13:53:05.696586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:05.695112    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 13:53:06.660704 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:06.660675    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 13:53:06.660861 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:06.660723    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 13:53:06.695888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:06.695856    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 13:53:06.697041 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:06.697011    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 13:53:07.660329 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:07.660292    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 13:53:07.660480 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:07.660342    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 13:53:07.700587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:07.698902    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 13:53:07.700587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:07.698979    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 13:53:08.659719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:08.659682    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 13:53:08.659902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:08.659733    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 13:53:08.696207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:08.695847    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 13:53:08.696207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:08.695899    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 13:53:09.660145 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:09.660102    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 13:53:09.660540 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:09.660171    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 13:53:09.694612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:09.694547    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 13:53:09.694771 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:09.694648    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 13:53:11.192641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:11.192605    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:11.193348 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:11.193329    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:11.194336 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:11.194317    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc"
Nov 02 13:53:11.226710 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:11.226690    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:11.226957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:11.226937    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:11.227879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:11.227857    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn"
Nov 02 13:53:12.188620 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.188594    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"]
Nov 02 13:53:12.188831 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.188787    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg" podUID="8c05a58d-1880-4b4e-9ee8-7030d42e5688" containerName="manager" containerID="cri-o://eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9" gracePeriod=30
Nov 02 13:53:12.431187 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.431163    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:53:12.530836 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.530766    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c05a58d-1880-4b4e-9ee8-7030d42e5688-kube-api-access-ghwvf\" (UniqueName: \"kubernetes.io/projected/8c05a58d-1880-4b4e-9ee8-7030d42e5688-kube-api-access-ghwvf\") pod \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") "
Nov 02 13:53:12.530836 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.530834    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c05a58d-1880-4b4e-9ee8-7030d42e5688-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c05a58d-1880-4b4e-9ee8-7030d42e5688-metrics-certs\") pod \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\" (UID: \"8c05a58d-1880-4b4e-9ee8-7030d42e5688\") "
Nov 02 13:53:12.532941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.532909    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c05a58d-1880-4b4e-9ee8-7030d42e5688-metrics-certs" pod "8c05a58d-1880-4b4e-9ee8-7030d42e5688" (UID: "8c05a58d-1880-4b4e-9ee8-7030d42e5688"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:53:12.533047 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.532984    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c05a58d-1880-4b4e-9ee8-7030d42e5688-kube-api-access-ghwvf" pod "8c05a58d-1880-4b4e-9ee8-7030d42e5688" (UID: "8c05a58d-1880-4b4e-9ee8-7030d42e5688"). InnerVolumeSpecName "kube-api-access-ghwvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:53:12.631478 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.631445    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghwvf\" (UniqueName: \"kubernetes.io/projected/8c05a58d-1880-4b4e-9ee8-7030d42e5688-kube-api-access-ghwvf\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:53:12.631478 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.631468    5572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c05a58d-1880-4b4e-9ee8-7030d42e5688-metrics-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:53:12.822780 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.822748    5572 generic.go:356] "Generic (PLEG): container finished" podID="8c05a58d-1880-4b4e-9ee8-7030d42e5688" containerID="eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9" exitCode=0
Nov 02 13:53:12.822942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.822802    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"
Nov 02 13:53:12.822942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.822824    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg" event={"ID":"8c05a58d-1880-4b4e-9ee8-7030d42e5688","Type":"ContainerDied","Data":"eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9"}
Nov 02 13:53:12.822942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.822877    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg" event={"ID":"8c05a58d-1880-4b4e-9ee8-7030d42e5688","Type":"ContainerDied","Data":"f105411265259fa86d09dc05bcee656a7c8e8ccc854121f52c13feb3a6b8f266"}
Nov 02 13:53:12.822942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.822898    5572 scope.go:122] "RemoveContainer" containerID="eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9"
Nov 02 13:53:12.834967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.834873    5572 scope.go:122] "RemoveContainer" containerID="eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9"
Nov 02 13:53:12.835353 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:53:12.835333    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9\": container with ID starting with eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9 not found: ID does not exist" containerID="eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9"
Nov 02 13:53:12.835431 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.835359    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9"} err="failed to get container status \"eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9\": rpc error: code = NotFound desc = could not find container \"eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9\": container with ID starting with eebe5ce429cd1759120d697138ea21764ca71872acd48ef4e693d818f375b7d9 not found: ID does not exist"
Nov 02 13:53:12.850704 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.850685    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"]
Nov 02 13:53:12.856601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:12.856580    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-rpvrg"]
Nov 02 13:53:13.541277 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:13.541235    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c05a58d-1880-4b4e-9ee8-7030d42e5688" path="/var/lib/kubelet/pods/8c05a58d-1880-4b4e-9ee8-7030d42e5688/volumes"
Nov 02 13:53:21.428925 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:53:21.428880    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:53:49.964293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:49.964217    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"]
Nov 02 13:53:49.965239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:49.965218    5572 cpu_manager.go:414] "containerMap: removing container" podUID="8c05a58d-1880-4b4e-9ee8-7030d42e5688" containerName="manager"
Nov 02 13:53:49.965239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:49.965238    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c05a58d-1880-4b4e-9ee8-7030d42e5688" containerName="manager"
Nov 02 13:53:49.965435 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:49.965342    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c05a58d-1880-4b4e-9ee8-7030d42e5688" containerName="manager"
Nov 02 13:53:49.969879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:49.969858    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:49.974214 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:49.974183    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9e0c2f-openshift-default-dockercfg-xxct7\""
Nov 02 13:53:49.994186 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:49.994163    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"]
Nov 02 13:53:50.069892 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.069865    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070020 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.069895    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070020 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.069929    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070020 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.069998    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8nnl\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-kube-api-access-b8nnl\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.070105    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070158 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.070141    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070209 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.070196    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.070218    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.070276 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.070241    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce05724-741f-45ba-bb1e-dce977afe40d-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171477 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171447    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171477 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171479    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171505    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171533    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nnl\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-kube-api-access-b8nnl\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171621    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171661    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171692    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.171736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171716    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.172035 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171750    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce05724-741f-45ba-bb1e-dce977afe40d-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.172035 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.171955    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.172035 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.172023    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.172323 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.172299    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.172439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.172422    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.172785 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.172764    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce05724-741f-45ba-bb1e-dce977afe40d-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.174019 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.173994    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.174225 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.174204    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.199253 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.199231    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nnl\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-kube-api-access-b8nnl\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.204869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.204844    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.219204 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.219145    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"]
Nov 02 13:53:50.223857 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.223841    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.231204 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.231185    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-1eb567-openshift-default-dockercfg-lpvp4\""
Nov 02 13:53:50.237769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.237750    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"]
Nov 02 13:53:50.285584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.285532    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:50.373930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.373892    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374077 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.373946    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374077 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.374039    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374195 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.374078    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bzpj\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-kube-api-access-2bzpj\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.374211    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374296 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.374255    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374371 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.374350    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374441 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.374427    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.374553 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.374529    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476186 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476115    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476186 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476175    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476201    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476218    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476233    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bzpj\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-kube-api-access-2bzpj\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476273    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476307    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476341    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.476642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476404    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.477022 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476922    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.477022 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.476996    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.477221 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.477095    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.477284 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.477268    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.482600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.480415    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.482600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.480951    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.488222 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.488195    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.498409 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.498385    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.498609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.498592    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bzpj\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-kube-api-access-2bzpj\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.537613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.537551    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:50.571504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.571479    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"]
Nov 02 13:53:50.576197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.576176    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.587353 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.587333    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f2e945-openshift-default-dockercfg-d2ckm\""
Nov 02 13:53:50.600989 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.597446    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"]
Nov 02 13:53:50.610232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.610208    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"]
Nov 02 13:53:50.615803 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.615768    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:50.615898 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.615854    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:50.615945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.615895    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:50.678957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.678932    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwh5j\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-kube-api-access-rwh5j\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679064 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.678991    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679129 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.679088    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679184 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.679152    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679285 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.679263    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679336 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.679298    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679336 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.679317    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679420 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.679404    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.679458 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.679431    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.780142 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.780071    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.780350 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.780331    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.780466 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.780447    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.780655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.780639    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.780779 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.780764    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.780907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.780891    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwh5j\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-kube-api-access-rwh5j\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.781055 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.781039    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.781167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.781151    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.781315 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.781297    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.781516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.781448    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.781516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.781098    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.781516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.781212    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.781821 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.781799    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.782366 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.782340    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.785322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.785301    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.787975 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.787926    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.794081 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.793990    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.800256 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.800194    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwh5j\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-kube-api-access-rwh5j\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.835964 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.835934    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"]
Nov 02 13:53:50.839817 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:50.839776    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b79d4c7_4fb0_4647_823a_e6a4f1cd13ad.slice/crio-47965254e9299ab2752ac765ba556f12b02d7e46daa6db7fb7f1963779b7ea8c WatchSource:0}: Error finding container 47965254e9299ab2752ac765ba556f12b02d7e46daa6db7fb7f1963779b7ea8c: Status 404 returned error can't find the container with id 47965254e9299ab2752ac765ba556f12b02d7e46daa6db7fb7f1963779b7ea8c
Nov 02 13:53:50.842288 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.842256    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:50.842392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.842328    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:50.842392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.842374    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:50.891309 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.891286    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:50.991611 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.991525    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" event={"ID":"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad","Type":"ContainerStarted","Data":"1eed0bedaaf4a84bb719ff5e896b7619944e2c109e2e65c6a34d1e55a8896df1"}
Nov 02 13:53:50.992090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.991647    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" event={"ID":"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad","Type":"ContainerStarted","Data":"47965254e9299ab2752ac765ba556f12b02d7e46daa6db7fb7f1963779b7ea8c"}
Nov 02 13:53:50.995791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.995766    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" event={"ID":"2ce05724-741f-45ba-bb1e-dce977afe40d","Type":"ContainerStarted","Data":"f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40"}
Nov 02 13:53:50.995923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:50.995806    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" event={"ID":"2ce05724-741f-45ba-bb1e-dce977afe40d","Type":"ContainerStarted","Data":"471821754c0cad07c30e70fec6e8dabed751b5d6e7bca136c43fc4bee7754b51"}
Nov 02 13:53:51.026784 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.026722    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podStartSLOduration=1.02670232 podStartE2EDuration="1.02670232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:51.022802178 +0000 UTC m=+607.600891046" watchObservedRunningTime="2026-11-02 13:53:51.02670232 +0000 UTC m=+607.604791187"
Nov 02 13:53:51.053245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.053149    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podStartSLOduration=2.053125342 podStartE2EDuration="2.053125342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:51.051752257 +0000 UTC m=+607.629841125" watchObservedRunningTime="2026-11-02 13:53:51.053125342 +0000 UTC m=+607.631214203"
Nov 02 13:53:51.189009 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.188960    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"]
Nov 02 13:53:51.200840 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:51.200799    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd26d6e3_9276_4d2a_8528_5ec79ac846cf.slice/crio-497f0847d9e0ecc0adc272f6c294cd6051d3edd8e9421e85933d58f59beee301 WatchSource:0}: Error finding container 497f0847d9e0ecc0adc272f6c294cd6051d3edd8e9421e85933d58f59beee301: Status 404 returned error can't find the container with id 497f0847d9e0ecc0adc272f6c294cd6051d3edd8e9421e85933d58f59beee301
Nov 02 13:53:51.206184 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.204161    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:51.206184 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.204250    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:51.206184 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.204293    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:53:51.285937 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.285898    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:51.292079 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.292056    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:51.457258 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:53:51.457214    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:53:51.543254 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.543231    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:51.543851 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:51.543827    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:52.002662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.002624    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" event={"ID":"dd26d6e3-9276-4d2a-8528-5ec79ac846cf","Type":"ContainerStarted","Data":"b013ec7780e1f9722b6c9e6c87e15b8d279b1612fae66e2bafde696c55926382"}
Nov 02 13:53:52.003042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.002693    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" event={"ID":"dd26d6e3-9276-4d2a-8528-5ec79ac846cf","Type":"ContainerStarted","Data":"497f0847d9e0ecc0adc272f6c294cd6051d3edd8e9421e85933d58f59beee301"}
Nov 02 13:53:52.003155 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.003139    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:52.003211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.003183    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:52.003941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.003918    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:53:52.004084 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.004062    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:53:52.042350 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.042312    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podStartSLOduration=2.042299716 podStartE2EDuration="2.042299716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:52.041156136 +0000 UTC m=+608.619245005" watchObservedRunningTime="2026-11-02 13:53:52.042299716 +0000 UTC m=+608.620388583"
Nov 02 13:53:52.891597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.891542    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:52.897030 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:52.897006    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:53.006991 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:53.006957    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:53.007749 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:53.007732    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:53:55.387058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.387028    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"]
Nov 02 13:53:55.390643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.390621    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:55.410063 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.410044    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9e0c2f-dockercfg-vrxzc\""
Nov 02 13:53:55.419135 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.419109    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"]
Nov 02 13:53:55.440368 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.440350    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9e0c2f\""
Nov 02 13:53:55.526473 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.526433    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/553121b1-c367-4a01-868c-e84169451981-kube-api-access-c2bcg\") pod \"payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:55.526473 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.526477    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553121b1-c367-4a01-868c-e84169451981-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:55.628103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.628071    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/553121b1-c367-4a01-868c-e84169451981-kube-api-access-c2bcg\") pod \"payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:55.628251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.628113    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553121b1-c367-4a01-868c-e84169451981-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:55.628942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.628923    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553121b1-c367-4a01-868c-e84169451981-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:55.644520 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.644463    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/553121b1-c367-4a01-868c-e84169451981-kube-api-access-c2bcg\") pod \"payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:55.705301 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:55.705272    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:56.012519 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:56.012439    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"]
Nov 02 13:53:56.015747 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:56.015720    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553121b1_c367_4a01_868c_e84169451981.slice/crio-b3b321898d2f662de0817cc1b5a09c2ccc301a0cd41c640858ee07a398e3293c WatchSource:0}: Error finding container b3b321898d2f662de0817cc1b5a09c2ccc301a0cd41c640858ee07a398e3293c: Status 404 returned error can't find the container with id b3b321898d2f662de0817cc1b5a09c2ccc301a0cd41c640858ee07a398e3293c
Nov 02 13:53:56.022627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:56.022599    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" event={"ID":"553121b1-c367-4a01-868c-e84169451981","Type":"ContainerStarted","Data":"b3b321898d2f662de0817cc1b5a09c2ccc301a0cd41c640858ee07a398e3293c"}
Nov 02 13:53:57.029607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.029553    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"]
Nov 02 13:53:57.033462 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.033438    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" event={"ID":"553121b1-c367-4a01-868c-e84169451981","Type":"ContainerStarted","Data":"ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49"}
Nov 02 13:53:57.033600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.033471    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:53:57.033600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.033546    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.048265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.048245    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4fc49b\""
Nov 02 13:53:57.063688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.063667    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"]
Nov 02 13:53:57.067198 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.067175    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"]
Nov 02 13:53:57.071031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.071013    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.084391 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.084328    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4fc49b-dockercfg-25jlx\""
Nov 02 13:53:57.087141 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.087101    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" podStartSLOduration=2.087089127 podStartE2EDuration="2.087089127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:57.084546208 +0000 UTC m=+613.662635067" watchObservedRunningTime="2026-11-02 13:53:57.087089127 +0000 UTC m=+613.665177995"
Nov 02 13:53:57.091984 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.091958    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"]
Nov 02 13:53:57.161934 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.161909    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp99b\" (UniqueName: \"kubernetes.io/projected/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-kube-api-access-dp99b\") pod \"payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.162083 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.161952    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.162156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.162118    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.162274 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.162254    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6xh\" (UniqueName: \"kubernetes.io/projected/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-kube-api-access-jn6xh\") pod \"payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.266582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.264177    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6xh\" (UniqueName: \"kubernetes.io/projected/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-kube-api-access-jn6xh\") pod \"payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.266582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.264257    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp99b\" (UniqueName: \"kubernetes.io/projected/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-kube-api-access-dp99b\") pod \"payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.266582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.264290    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.266582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.264392    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.266582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.265356    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.266582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.265994    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.286437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.286364    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp99b\" (UniqueName: \"kubernetes.io/projected/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-kube-api-access-dp99b\") pod \"payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.286603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.286372    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6xh\" (UniqueName: \"kubernetes.io/projected/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-kube-api-access-jn6xh\") pod \"payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.346416 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.346390    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:57.384737 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.384712    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:57.628144 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.628118    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"]
Nov 02 13:53:57.630342 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:57.630316    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12c7cb2_83e6_49ef_824b_c75f61bbd77f.slice/crio-0591b8bb303f9082c4d103094022548b68e59a174b3a4db042e3df8528e9689b WatchSource:0}: Error finding container 0591b8bb303f9082c4d103094022548b68e59a174b3a4db042e3df8528e9689b: Status 404 returned error can't find the container with id 0591b8bb303f9082c4d103094022548b68e59a174b3a4db042e3df8528e9689b
Nov 02 13:53:57.699224 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:57.696207    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38391be5_0a0a_486d_b9a8_a6b9e4da6e93.slice/crio-977634c4d5cd52cc58d6a42c2e6dcd19df07c7f6fe4a14fa9df9d293c88e7124 WatchSource:0}: Error finding container 977634c4d5cd52cc58d6a42c2e6dcd19df07c7f6fe4a14fa9df9d293c88e7124: Status 404 returned error can't find the container with id 977634c4d5cd52cc58d6a42c2e6dcd19df07c7f6fe4a14fa9df9d293c88e7124
Nov 02 13:53:57.699224 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:57.699120    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"]
Nov 02 13:53:58.037838 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.037760    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" event={"ID":"e12c7cb2-83e6-49ef-824b-c75f61bbd77f","Type":"ContainerStarted","Data":"78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4"}
Nov 02 13:53:58.037838 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.037816    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" event={"ID":"e12c7cb2-83e6-49ef-824b-c75f61bbd77f","Type":"ContainerStarted","Data":"0591b8bb303f9082c4d103094022548b68e59a174b3a4db042e3df8528e9689b"}
Nov 02 13:53:58.037838 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.037841    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:53:58.039325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.039302    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" event={"ID":"38391be5-0a0a-486d-b9a8-a6b9e4da6e93","Type":"ContainerStarted","Data":"526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38"}
Nov 02 13:53:58.039430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.039340    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" event={"ID":"38391be5-0a0a-486d-b9a8-a6b9e4da6e93","Type":"ContainerStarted","Data":"977634c4d5cd52cc58d6a42c2e6dcd19df07c7f6fe4a14fa9df9d293c88e7124"}
Nov 02 13:53:58.039430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.039390    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:53:58.058495 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.058453    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" podStartSLOduration=2.058439185 podStartE2EDuration="2.058439185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:58.056267103 +0000 UTC m=+614.634355973" watchObservedRunningTime="2026-11-02 13:53:58.058439185 +0000 UTC m=+614.636528056"
Nov 02 13:53:58.075369 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.075333    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" podStartSLOduration=1.075323285 podStartE2EDuration="1.075323285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:58.074154637 +0000 UTC m=+614.652243516" watchObservedRunningTime="2026-11-02 13:53:58.075323285 +0000 UTC m=+614.653412152"
Nov 02 13:53:58.316504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.316474    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"]
Nov 02 13:53:58.321110 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.321090    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.332434 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.332413    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-1d18fb\""
Nov 02 13:53:58.335169 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.335147    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"]
Nov 02 13:53:58.350603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.350586    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-1d18fb-dockercfg-k4c4p\""
Nov 02 13:53:58.480180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.480132    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a012951-d303-4275-b064-d61ff42b0699-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.480344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.480181    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czht\" (UniqueName: \"kubernetes.io/projected/5a012951-d303-4275-b064-d61ff42b0699-kube-api-access-6czht\") pod \"payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.581502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.581423    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a012951-d303-4275-b064-d61ff42b0699-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.581502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.581454    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czht\" (UniqueName: \"kubernetes.io/projected/5a012951-d303-4275-b064-d61ff42b0699-kube-api-access-6czht\") pod \"payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.582098 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.582080    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a012951-d303-4275-b064-d61ff42b0699-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.593686 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.593661    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czht\" (UniqueName: \"kubernetes.io/projected/5a012951-d303-4275-b064-d61ff42b0699-kube-api-access-6czht\") pod \"payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.636277 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.636252    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:58.934356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:58.934328    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"]
Nov 02 13:53:58.937227 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:58.937194    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a012951_d303_4275_b064_d61ff42b0699.slice/crio-9579de7f7cf9f299931c85cc8812e27ac768b448dc5f7ee107aa7745e032ecea WatchSource:0}: Error finding container 9579de7f7cf9f299931c85cc8812e27ac768b448dc5f7ee107aa7745e032ecea: Status 404 returned error can't find the container with id 9579de7f7cf9f299931c85cc8812e27ac768b448dc5f7ee107aa7745e032ecea
Nov 02 13:53:59.046533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.046499    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" event={"ID":"5a012951-d303-4275-b064-d61ff42b0699","Type":"ContainerStarted","Data":"fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f"}
Nov 02 13:53:59.046965 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.046891    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:53:59.046965 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.046934    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" event={"ID":"5a012951-d303-4275-b064-d61ff42b0699","Type":"ContainerStarted","Data":"9579de7f7cf9f299931c85cc8812e27ac768b448dc5f7ee107aa7745e032ecea"}
Nov 02 13:53:59.068320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.068274    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" podStartSLOduration=1.06826048 podStartE2EDuration="1.06826048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:59.0666452 +0000 UTC m=+615.644734081" watchObservedRunningTime="2026-11-02 13:53:59.06826048 +0000 UTC m=+615.646349362"
Nov 02 13:53:59.094939 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.094908    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"]
Nov 02 13:53:59.100280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.100251    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.114656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.114632    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ee7df2-dockercfg-f42bp\""
Nov 02 13:53:59.114798 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.114766    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ee7df2\""
Nov 02 13:53:59.114930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.114912    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"]
Nov 02 13:53:59.200786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.200661    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4f40f-eae5-4eed-8903-7019fb0cf419-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.202148 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.202122    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2hd\" (UniqueName: \"kubernetes.io/projected/0cf4f40f-eae5-4eed-8903-7019fb0cf419-kube-api-access-jh2hd\") pod \"payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.304917 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.304887    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2hd\" (UniqueName: \"kubernetes.io/projected/0cf4f40f-eae5-4eed-8903-7019fb0cf419-kube-api-access-jh2hd\") pod \"payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.305089 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.304986    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4f40f-eae5-4eed-8903-7019fb0cf419-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.305726 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.305709    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4f40f-eae5-4eed-8903-7019fb0cf419-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.318847 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.318821    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2hd\" (UniqueName: \"kubernetes.io/projected/0cf4f40f-eae5-4eed-8903-7019fb0cf419-kube-api-access-jh2hd\") pod \"payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.413912 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.413882    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:53:59.586633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.586597    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"]
Nov 02 13:53:59.591431 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.591413    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:53:59.600911 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.600865    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"]
Nov 02 13:53:59.607223 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.607200    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7a9443-dockercfg-tj2w2\""
Nov 02 13:53:59.607358 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.607346    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7a9443\""
Nov 02 13:53:59.708705 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.708677    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d015852-ebcd-4cc8-9e80-6a025a410d7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:53:59.708879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.708716    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlw9\" (UniqueName: \"kubernetes.io/projected/3d015852-ebcd-4cc8-9e80-6a025a410d7a-kube-api-access-9hlw9\") pod \"payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:53:59.709347 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.709317    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"]
Nov 02 13:53:59.712270 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:53:59.712232    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf4f40f_eae5_4eed_8903_7019fb0cf419.slice/crio-d71be863854a40ced6fc5291871f98e0fe1e0af40c5c7281bff63ffe53f56164 WatchSource:0}: Error finding container d71be863854a40ced6fc5291871f98e0fe1e0af40c5c7281bff63ffe53f56164: Status 404 returned error can't find the container with id d71be863854a40ced6fc5291871f98e0fe1e0af40c5c7281bff63ffe53f56164
Nov 02 13:53:59.811265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.810735    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d015852-ebcd-4cc8-9e80-6a025a410d7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:53:59.811265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.810844    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlw9\" (UniqueName: \"kubernetes.io/projected/3d015852-ebcd-4cc8-9e80-6a025a410d7a-kube-api-access-9hlw9\") pod \"payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:53:59.811708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.811677    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d015852-ebcd-4cc8-9e80-6a025a410d7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:53:59.855584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.852264    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlw9\" (UniqueName: \"kubernetes.io/projected/3d015852-ebcd-4cc8-9e80-6a025a410d7a-kube-api-access-9hlw9\") pod \"payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:53:59.908170 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:53:59.908135    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:54:00.053911 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.053835    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" event={"ID":"0cf4f40f-eae5-4eed-8903-7019fb0cf419","Type":"ContainerStarted","Data":"46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613"}
Nov 02 13:54:00.053911 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.053893    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" event={"ID":"0cf4f40f-eae5-4eed-8903-7019fb0cf419","Type":"ContainerStarted","Data":"d71be863854a40ced6fc5291871f98e0fe1e0af40c5c7281bff63ffe53f56164"}
Nov 02 13:54:00.054346 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.054057    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:54:00.081280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.081234    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" podStartSLOduration=1.081222862 podStartE2EDuration="1.081222862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:00.079867647 +0000 UTC m=+616.657956516" watchObservedRunningTime="2026-11-02 13:54:00.081222862 +0000 UTC m=+616.659311729"
Nov 02 13:54:00.189237 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.189151    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"]
Nov 02 13:54:00.191282 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:54:00.191253    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d015852_ebcd_4cc8_9e80_6a025a410d7a.slice/crio-fec59d6f84b1a5d92f9e086012bb3597c4cf9ffb269411e6dde1f762e0aef344 WatchSource:0}: Error finding container fec59d6f84b1a5d92f9e086012bb3597c4cf9ffb269411e6dde1f762e0aef344: Status 404 returned error can't find the container with id fec59d6f84b1a5d92f9e086012bb3597c4cf9ffb269411e6dde1f762e0aef344
Nov 02 13:54:00.312178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.312153    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"]
Nov 02 13:54:00.315787 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.315770    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.322901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.322883    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f2e945-dockercfg-g8fv9\""
Nov 02 13:54:00.323009 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.322964    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f2e945\""
Nov 02 13:54:00.325772 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.325712    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"]
Nov 02 13:54:00.416776 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.416748    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb47d\" (UniqueName: \"kubernetes.io/projected/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-kube-api-access-rb47d\") pod \"payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.416901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.416813    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.517768 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.517736    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb47d\" (UniqueName: \"kubernetes.io/projected/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-kube-api-access-rb47d\") pod \"payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.517935 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.517786    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.518413 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.518394    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.532826 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.532792    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb47d\" (UniqueName: \"kubernetes.io/projected/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-kube-api-access-rb47d\") pod \"payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.630910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.630848    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:00.948807 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:00.946658    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"]
Nov 02 13:54:00.951127 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:54:00.951061    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bf1d4a_09de_4b3b_a86f_d6a04d8f0bd0.slice/crio-1bd3789a01e98349d6fc8fab9f6db49330e13e666bcc6459aec872bed16c353a WatchSource:0}: Error finding container 1bd3789a01e98349d6fc8fab9f6db49330e13e666bcc6459aec872bed16c353a: Status 404 returned error can't find the container with id 1bd3789a01e98349d6fc8fab9f6db49330e13e666bcc6459aec872bed16c353a
Nov 02 13:54:01.065535 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:01.065504    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" event={"ID":"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0","Type":"ContainerStarted","Data":"e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8"}
Nov 02 13:54:01.065901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:01.065576    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" event={"ID":"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0","Type":"ContainerStarted","Data":"1bd3789a01e98349d6fc8fab9f6db49330e13e666bcc6459aec872bed16c353a"}
Nov 02 13:54:01.066197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:01.066171    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:01.067648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:01.067621    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" event={"ID":"3d015852-ebcd-4cc8-9e80-6a025a410d7a","Type":"ContainerStarted","Data":"a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81"}
Nov 02 13:54:01.067758 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:01.067668    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" event={"ID":"3d015852-ebcd-4cc8-9e80-6a025a410d7a","Type":"ContainerStarted","Data":"fec59d6f84b1a5d92f9e086012bb3597c4cf9ffb269411e6dde1f762e0aef344"}
Nov 02 13:54:01.067758 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:01.067694    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:54:01.118583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:01.106146    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" podStartSLOduration=1.106127484 podStartE2EDuration="1.106127484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:01.100057507 +0000 UTC m=+617.678146373" watchObservedRunningTime="2026-11-02 13:54:01.106127484 +0000 UTC m=+617.684216352"
Nov 02 13:54:08.043549 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:08.043517    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 13:54:08.044708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:08.044673    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:54:08.044838 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:08.044722    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:54:08.044838 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:08.044795    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:54:08.044921 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:08.044841    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:54:08.079770 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:08.079720    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" podStartSLOduration=9.079706866 podStartE2EDuration="9.079706866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:01.13288432 +0000 UTC m=+617.710973188" watchObservedRunningTime="2026-11-02 13:54:08.079706866 +0000 UTC m=+624.657795730"
Nov 02 13:54:09.049852 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:09.049821    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 13:54:09.050441 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:09.049886    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 13:54:10.058616 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.057908    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:54:10.169962 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.169931    5572 generic.go:356] "Generic (PLEG): container finished" podID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerID="1eed0bedaaf4a84bb719ff5e896b7619944e2c109e2e65c6a34d1e55a8896df1" exitCode=0
Nov 02 13:54:10.170130 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.170004    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" event={"ID":"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad","Type":"ContainerDied","Data":"1eed0bedaaf4a84bb719ff5e896b7619944e2c109e2e65c6a34d1e55a8896df1"}
Nov 02 13:54:10.170470 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.170451    5572 scope.go:122] "RemoveContainer" containerID="1eed0bedaaf4a84bb719ff5e896b7619944e2c109e2e65c6a34d1e55a8896df1"
Nov 02 13:54:10.171805 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.171783    5572 generic.go:356] "Generic (PLEG): container finished" podID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerID="f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40" exitCode=0
Nov 02 13:54:10.171899 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.171843    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" event={"ID":"2ce05724-741f-45ba-bb1e-dce977afe40d","Type":"ContainerDied","Data":"f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40"}
Nov 02 13:54:10.171950 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.171888    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:10.171983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.171953    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:10.172016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.171992    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:10.172153 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.172141    5572 scope.go:122] "RemoveContainer" containerID="f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40"
Nov 02 13:54:10.173418 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.173389    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:10.173486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.173468    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:10.173527 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:10.173514    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:11.072204 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.071542    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 13:54:11.179087 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.179061    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" event={"ID":"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad","Type":"ContainerStarted","Data":"97d38eb55cedafbc88770b89a03ac71334b8dd58521ff00b712816563ef786fd"}
Nov 02 13:54:11.206627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.190386    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" event={"ID":"2ce05724-741f-45ba-bb1e-dce977afe40d","Type":"ContainerStarted","Data":"bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e"}
Nov 02 13:54:11.305590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.303961    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:54:11.305590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.304118    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:54:11.305590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.304161    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 13:54:11.538205 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.538109    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 13:54:11.538205 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.538183    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 13:54:11.543044 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:11.543021    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:54:12.075522 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:12.075498    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 13:54:12.292584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:12.288946    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:54:12.292584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:12.289003    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 13:54:12.543642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:12.543318    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 13:54:12.543642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:12.543380    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 13:54:13.081578 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:13.080692    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 13:54:13.292589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:13.286070    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:54:13.292589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:13.286129    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 13:54:13.571114 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:13.571075    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 13:54:13.571314 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:13.571151    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 13:54:14.288584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:14.287930    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:54:14.288584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:14.287991    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 13:54:14.544640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:14.543922    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 13:54:14.544640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:14.543982    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 13:54:15.287548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:15.287515    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:54:15.287871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:15.287819    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 13:54:15.548637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:15.543807    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 13:54:15.548637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:15.543864    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 13:54:16.287137 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:16.287093    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:54:16.287465 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:16.287417    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 13:54:16.614959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:16.614923    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:54:16.616777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:16.616751    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:54:16.617720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:16.617701    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 13:54:17.311889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:17.311856    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:54:17.312521 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:17.312485    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:54:17.314137 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:17.314115    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 13:54:18.125821 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.125793    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"]
Nov 02 13:54:18.129092 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.129073    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.138911 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.138889    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"]
Nov 02 13:54:18.140396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.140375    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4fc49b\"/\"kube-root-ca.crt\""
Nov 02 13:54:18.140606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.140585    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4fc49b\"/\"simulator-w3-4fc49b-kserve-self-signed-certs\""
Nov 02 13:54:18.140723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.140681    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4fc49b\"/\"openshift-service-ca.crt\""
Nov 02 13:54:18.140781 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.140769    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4fc49b\"/\"default-dockercfg-9pj54\""
Nov 02 13:54:18.265628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.265597    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34030298-d188-4419-926b-d865e0618a80-tls-certs\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.265791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.265656    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-home\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.265791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.265739    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-tmp-dir\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.265902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.265859    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-model-cache\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.265902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.265885    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-dshm\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.266006 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.265965    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5ps\" (UniqueName: \"kubernetes.io/projected/34030298-d188-4419-926b-d865e0618a80-kube-api-access-mn5ps\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.369265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.369226    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34030298-d188-4419-926b-d865e0618a80-tls-certs\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.369499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.369326    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-home\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.369499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.369367    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-tmp-dir\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.369499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.369435    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-model-cache\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.369499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.369463    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-dshm\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.369819 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.369523    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5ps\" (UniqueName: \"kubernetes.io/projected/34030298-d188-4419-926b-d865e0618a80-kube-api-access-mn5ps\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.370212 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.370140    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-tmp-dir\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.370486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.370144    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-model-cache\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.370744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.370720    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-home\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.373430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.373401    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34030298-d188-4419-926b-d865e0618a80-tls-certs\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.373871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.373849    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-dshm\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.386594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.386501    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5ps\" (UniqueName: \"kubernetes.io/projected/34030298-d188-4419-926b-d865e0618a80-kube-api-access-mn5ps\") pod \"simulator-w3-4fc49b-kserve-6944c6d8d8-jj969\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") " pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.443452 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.443417    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:18.719386 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:18.719364    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"]
Nov 02 13:54:18.722353 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:54:18.722322    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34030298_d188_4419_926b_d865e0618a80.slice/crio-34fe20b8cdb37a8f9d4388dd7c1ce114ee9b433f38a567a66c8ecc4d5bfb323e WatchSource:0}: Error finding container 34fe20b8cdb37a8f9d4388dd7c1ce114ee9b433f38a567a66c8ecc4d5bfb323e: Status 404 returned error can't find the container with id 34fe20b8cdb37a8f9d4388dd7c1ce114ee9b433f38a567a66c8ecc4d5bfb323e
Nov 02 13:54:19.349314 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:19.349278    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" event={"ID":"34030298-d188-4419-926b-d865e0618a80","Type":"ContainerStarted","Data":"715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d"}
Nov 02 13:54:19.349822 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:19.349323    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" event={"ID":"34030298-d188-4419-926b-d865e0618a80","Type":"ContainerStarted","Data":"34fe20b8cdb37a8f9d4388dd7c1ce114ee9b433f38a567a66c8ecc4d5bfb323e"}
Nov 02 13:54:19.389011 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:19.388963    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podStartSLOduration=1.388950417 podStartE2EDuration="1.388950417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:19.387955406 +0000 UTC m=+635.966044273" watchObservedRunningTime="2026-11-02 13:54:19.388950417 +0000 UTC m=+635.967039283"
Nov 02 13:54:19.444552 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:19.444527    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:21.360695 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:21.360662    5572 generic.go:356] "Generic (PLEG): container finished" podID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerID="b013ec7780e1f9722b6c9e6c87e15b8d279b1612fae66e2bafde696c55926382" exitCode=0
Nov 02 13:54:21.361150 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:21.360735    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" event={"ID":"dd26d6e3-9276-4d2a-8528-5ec79ac846cf","Type":"ContainerDied","Data":"b013ec7780e1f9722b6c9e6c87e15b8d279b1612fae66e2bafde696c55926382"}
Nov 02 13:54:21.361248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:21.361231    5572 scope.go:122] "RemoveContainer" containerID="b013ec7780e1f9722b6c9e6c87e15b8d279b1612fae66e2bafde696c55926382"
Nov 02 13:54:21.362383 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:21.362356    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:21.362486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:21.362434    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:21.362486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:21.362477    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:54:21.433703 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:54:21.433670    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:54:22.033690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.033661    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"]
Nov 02 13:54:22.036300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.036285    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.048225 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.048206    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d18fb\"/\"simulator-w5-1d18fb-kserve-self-signed-certs\""
Nov 02 13:54:22.048378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.048363    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d18fb\"/\"default-dockercfg-64xsj\""
Nov 02 13:54:22.052471 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.052455    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1d18fb\"/\"openshift-service-ca.crt\""
Nov 02 13:54:22.056062 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.056050    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1d18fb\"/\"kube-root-ca.crt\""
Nov 02 13:54:22.066719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.066700    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"]
Nov 02 13:54:22.114073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.114039    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-dshm\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.114249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.114083    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tmp-dir\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.114249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.114158    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-model-cache\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.114249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.114178    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz88b\" (UniqueName: \"kubernetes.io/projected/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-kube-api-access-hz88b\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.114249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.114197    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-home\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.114249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.114210    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tls-certs\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215393 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215355    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-dshm\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215393 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215396    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tmp-dir\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215425    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-model-cache\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215444    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz88b\" (UniqueName: \"kubernetes.io/projected/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-kube-api-access-hz88b\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215462    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-home\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215484    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tls-certs\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215851    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-model-cache\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215872    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tmp-dir\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.215991 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.215937    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-home\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.217799 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.217775    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-dshm\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.218100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.218083    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tls-certs\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.231364 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.231341    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz88b\" (UniqueName: \"kubernetes.io/projected/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-kube-api-access-hz88b\") pod \"simulator-w5-1d18fb-kserve-7885c684c6-r6sq5\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") " pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.346744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.346712    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:22.368153 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.368127    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" event={"ID":"dd26d6e3-9276-4d2a-8528-5ec79ac846cf","Type":"ContainerStarted","Data":"02a7552ca953f9dddd09cc8bc18f31b78f567facb409e109b16c5adea230812e"}
Nov 02 13:54:22.650471 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.650447    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"]
Nov 02 13:54:22.652336 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:54:22.652305    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd70e1a5e_b2a0_4a0f_a0f8_693697f0dce6.slice/crio-761ae95c5fae1e1efd8d2dbee522bdb219d2909e8de6996a427a3478aa56cd79 WatchSource:0}: Error finding container 761ae95c5fae1e1efd8d2dbee522bdb219d2909e8de6996a427a3478aa56cd79: Status 404 returned error can't find the container with id 761ae95c5fae1e1efd8d2dbee522bdb219d2909e8de6996a427a3478aa56cd79
Nov 02 13:54:22.898584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.894852    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:54:22.898584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.895865    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 13:54:22.898584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:22.895913    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 13:54:23.006840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:23.006733    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:54:23.382578 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:23.382473    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" event={"ID":"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6","Type":"ContainerStarted","Data":"0aaa7e67c4abd5256d856e2d7af88e47e87bb0103b09af41ffc72f003a44c1a9"}
Nov 02 13:54:23.382578 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:23.382520    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" event={"ID":"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6","Type":"ContainerStarted","Data":"761ae95c5fae1e1efd8d2dbee522bdb219d2909e8de6996a427a3478aa56cd79"}
Nov 02 13:54:23.422601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:23.422530    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podStartSLOduration=2.422515554 podStartE2EDuration="2.422515554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:23.420769087 +0000 UTC m=+639.998857954" watchObservedRunningTime="2026-11-02 13:54:23.422515554 +0000 UTC m=+640.000604419"
Nov 02 13:54:23.905591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:23.904918    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 13:54:23.905591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:23.904977    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 13:54:24.347016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:24.346982    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:24.895125 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:24.895088    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 13:54:24.895759 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:24.895715    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 13:54:25.906710 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:25.906670    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 13:54:25.907218 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:25.906721    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 13:54:26.434802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.434773    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"]
Nov 02 13:54:26.437331 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.437315    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.457623 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.457596    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"]
Nov 02 13:54:26.457813 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.457797    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a9443\"/\"default-dockercfg-87xqh\""
Nov 02 13:54:26.457878 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.457867    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7a9443\"/\"simulator-w0-7a9443-kserve-self-signed-certs\""
Nov 02 13:54:26.457920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.457915    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7a9443\"/\"kube-root-ca.crt\""
Nov 02 13:54:26.457960 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.457951    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7a9443\"/\"openshift-service-ca.crt\""
Nov 02 13:54:26.565701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.565667    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-model-cache\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.565869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.565711    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tls-certs\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.565869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.565733    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tmp-dir\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.565869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.565822    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-home\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.565988 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.565883    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-dshm\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.565988 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.565917    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxjm5\" (UniqueName: \"kubernetes.io/projected/d7b3588b-c669-447e-9bd6-f07c5a8298d0-kube-api-access-bxjm5\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667225    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-dshm\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667265    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxjm5\" (UniqueName: \"kubernetes.io/projected/d7b3588b-c669-447e-9bd6-f07c5a8298d0-kube-api-access-bxjm5\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667391    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-model-cache\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667429    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tls-certs\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667595 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667462    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tmp-dir\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667595 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667496    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-home\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667814 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667793    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-model-cache\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667864    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-home\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.667977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.667953    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tmp-dir\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.669626 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.669601    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-dshm\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.670002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.669986    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tls-certs\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.680401 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.680381    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxjm5\" (UniqueName: \"kubernetes.io/projected/d7b3588b-c669-447e-9bd6-f07c5a8298d0-kube-api-access-bxjm5\") pod \"simulator-w0-7a9443-kserve-6656cf55c9-5l2vd\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") " pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.749112 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.749025    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:26.906023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.905987    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 13:54:26.906172 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:26.906047    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 13:54:27.067921 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:27.067896    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"]
Nov 02 13:54:27.070020 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:54:27.069995    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b3588b_c669_447e_9bd6_f07c5a8298d0.slice/crio-4b45f9dd9838292404f74280b2b47708fe9e325797d098ed2f450072a6b8ba42 WatchSource:0}: Error finding container 4b45f9dd9838292404f74280b2b47708fe9e325797d098ed2f450072a6b8ba42: Status 404 returned error can't find the container with id 4b45f9dd9838292404f74280b2b47708fe9e325797d098ed2f450072a6b8ba42
Nov 02 13:54:27.425818 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:27.425738    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" event={"ID":"d7b3588b-c669-447e-9bd6-f07c5a8298d0","Type":"ContainerStarted","Data":"fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd"}
Nov 02 13:54:27.425818 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:27.425773    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" event={"ID":"d7b3588b-c669-447e-9bd6-f07c5a8298d0","Type":"ContainerStarted","Data":"4b45f9dd9838292404f74280b2b47708fe9e325797d098ed2f450072a6b8ba42"}
Nov 02 13:54:27.481133 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:27.481065    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podStartSLOduration=1.481049085 podStartE2EDuration="1.481049085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:27.480821254 +0000 UTC m=+644.058910120" watchObservedRunningTime="2026-11-02 13:54:27.481049085 +0000 UTC m=+644.059137950"
Nov 02 13:54:27.749719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:27.749643    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:27.896074 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:27.896048    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:54:27.897063 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:27.897046    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 13:54:28.443845 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:28.443815    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:28.456592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:28.456547    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:29.457220 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:29.457186    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:54:32.354959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:32.350938    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:32.392656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:32.391911    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:32.468590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:32.468549    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:54:36.749958 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:36.749923    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:36.762887 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:36.762864    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:37.494682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:54:37.494645    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:54:51.429385 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:54:51.429350    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:55:01.352873 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.352835    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"]
Nov 02 13:55:01.356657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.356636    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.365082 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.365037    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1eb567\"/\"openshift-service-ca.crt\""
Nov 02 13:55:01.365280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.365152    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1eb567\"/\"premium-w6-1eb567-kserve-self-signed-certs\""
Nov 02 13:55:01.365399 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.365206    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1eb567\"/\"default-dockercfg-2r796\""
Nov 02 13:55:01.365491 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.365245    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1eb567\"/\"kube-root-ca.crt\""
Nov 02 13:55:01.367405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.367380    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"]
Nov 02 13:55:01.383476 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.383440    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-home\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.383624 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.383504    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-dshm\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.383693 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.383666    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdswp\" (UniqueName: \"kubernetes.io/projected/6c9c6a65-8430-434d-bc56-61b9afec9731-kube-api-access-bdswp\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.383740 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.383727    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-tmp-dir\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.383812 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.383794    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9c6a65-8430-434d-bc56-61b9afec9731-tls-certs\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.383854 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.383829    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-model-cache\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.500295 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.500243    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdswp\" (UniqueName: \"kubernetes.io/projected/6c9c6a65-8430-434d-bc56-61b9afec9731-kube-api-access-bdswp\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.500494 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.500363    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-tmp-dir\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.500494 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.500418    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9c6a65-8430-434d-bc56-61b9afec9731-tls-certs\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.500494 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.500458    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-model-cache\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.500716 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.500520    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-home\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.500716 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.500600    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-dshm\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.504234 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.504202    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-model-cache\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.514886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.505416    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-home\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.514886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.505657    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-tmp-dir\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.514886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.514475    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-dshm\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.514886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.514669    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9c6a65-8430-434d-bc56-61b9afec9731-tls-certs\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.525752 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.525721    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdswp\" (UniqueName: \"kubernetes.io/projected/6c9c6a65-8430-434d-bc56-61b9afec9731-kube-api-access-bdswp\") pod \"premium-w6-1eb567-kserve-54859757dd-29455\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") " pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.671122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.671046    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:01.948929 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:01.948844    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"]
Nov 02 13:55:01.951855 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:01.951824    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c9c6a65_8430_434d_bc56_61b9afec9731.slice/crio-2a329b9325b86c518147f5e75af77a79cb842d80f5a1902225c03fd5280d38d6 WatchSource:0}: Error finding container 2a329b9325b86c518147f5e75af77a79cb842d80f5a1902225c03fd5280d38d6: Status 404 returned error can't find the container with id 2a329b9325b86c518147f5e75af77a79cb842d80f5a1902225c03fd5280d38d6
Nov 02 13:55:02.629490 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:02.629452    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" event={"ID":"6c9c6a65-8430-434d-bc56-61b9afec9731","Type":"ContainerStarted","Data":"2f40fdf3ab5d615dafbfc3e753fb3a1be6c1655c59a777f09c02468e5c153e45"}
Nov 02 13:55:02.629920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:02.629504    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" event={"ID":"6c9c6a65-8430-434d-bc56-61b9afec9731","Type":"ContainerStarted","Data":"2a329b9325b86c518147f5e75af77a79cb842d80f5a1902225c03fd5280d38d6"}
Nov 02 13:55:02.652546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:02.652492    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podStartSLOduration=1.652476301 podStartE2EDuration="1.652476301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:02.651082517 +0000 UTC m=+679.229171385" watchObservedRunningTime="2026-11-02 13:55:02.652476301 +0000 UTC m=+679.230565167"
Nov 02 13:55:02.671921 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:02.671887    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:03.894144 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:03.894111    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"]
Nov 02 13:55:03.897971 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:03.897950    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:03.908344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:03.908322    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f2e945\"/\"default-dockercfg-ffm7d\""
Nov 02 13:55:03.908477 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:03.908323    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f2e945\"/\"premium-w1-f2e945-kserve-self-signed-certs\""
Nov 02 13:55:03.909169 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:03.909150    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"]
Nov 02 13:55:03.909885 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:03.909866    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f2e945\"/\"kube-root-ca.crt\""
Nov 02 13:55:03.910029 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:03.909958    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f2e945\"/\"openshift-service-ca.crt\""
Nov 02 13:55:04.038049 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.037977    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5zs\" (UniqueName: \"kubernetes.io/projected/dc5fe786-8a48-4a93-9be5-e2092ee937a5-kube-api-access-bp5zs\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.038049 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.038041    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-dshm\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.038294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.038085    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tmp-dir\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.038294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.038123    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-model-cache\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.038294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.038149    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-home\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.038294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.038201    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tls-certs\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.139957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.139912    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5zs\" (UniqueName: \"kubernetes.io/projected/dc5fe786-8a48-4a93-9be5-e2092ee937a5-kube-api-access-bp5zs\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.139969    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-dshm\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.139996    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tmp-dir\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.140026    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-model-cache\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.140044    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-home\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.140082    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tls-certs\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140511 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.140487    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tmp-dir\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140630 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.140549    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-home\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.140630 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.140617    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-model-cache\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.142694 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.142673    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-dshm\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.142848 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.142831    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tls-certs\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.161874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.161799    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5zs\" (UniqueName: \"kubernetes.io/projected/dc5fe786-8a48-4a93-9be5-e2092ee937a5-kube-api-access-bp5zs\") pod \"premium-w1-f2e945-kserve-55db7d458b-drszk\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") " pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.212456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.212417    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:04.492924 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.492900    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"]
Nov 02 13:55:04.494960 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:04.494938    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc5fe786_8a48_4a93_9be5_e2092ee937a5.slice/crio-b84a94acc72a07248c8f2000fe7c182c767c41c52369163c6f24ee2fafadc725 WatchSource:0}: Error finding container b84a94acc72a07248c8f2000fe7c182c767c41c52369163c6f24ee2fafadc725: Status 404 returned error can't find the container with id b84a94acc72a07248c8f2000fe7c182c767c41c52369163c6f24ee2fafadc725
Nov 02 13:55:04.642512 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.642461    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" event={"ID":"dc5fe786-8a48-4a93-9be5-e2092ee937a5","Type":"ContainerStarted","Data":"15399b7e8076dfdc465b07db73f51e324bc6d32e5be6478ab7443ca1b681a249"}
Nov 02 13:55:04.642673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.642524    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" event={"ID":"dc5fe786-8a48-4a93-9be5-e2092ee937a5","Type":"ContainerStarted","Data":"b84a94acc72a07248c8f2000fe7c182c767c41c52369163c6f24ee2fafadc725"}
Nov 02 13:55:04.672946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:04.672897    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podStartSLOduration=1.672883066 podStartE2EDuration="1.672883066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:04.671955143 +0000 UTC m=+681.250044017" watchObservedRunningTime="2026-11-02 13:55:04.672883066 +0000 UTC m=+681.250971947"
Nov 02 13:55:05.213385 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.213358    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:05.665839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.665809    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"]
Nov 02 13:55:05.669364 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.669348    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.674521 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.674502    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1d18fb-openshift-default-dockercfg-j42rt\""
Nov 02 13:55:05.683201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.683179    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"]
Nov 02 13:55:05.761424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761397    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761533    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761578    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761733 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761713    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c210602-64e7-4655-b4f6-102c28acca45-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761765    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761815    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761842    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c210602-64e7-4655-b4f6-102c28acca45-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761857    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96jzp\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-kube-api-access-96jzp\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.761925 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.761877    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863452 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863424    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c210602-64e7-4655-b4f6-102c28acca45-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863463    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863494    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863513    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c210602-64e7-4655-b4f6-102c28acca45-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863536    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96jzp\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-kube-api-access-96jzp\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863595    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863655    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863757    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.863907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863778    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.864056 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.863972    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.864056 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.864029    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.864156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.864119    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.864274 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.864254    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c210602-64e7-4655-b4f6-102c28acca45-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.864480 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.864459    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.866088 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.866066    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.866239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.866212    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c210602-64e7-4655-b4f6-102c28acca45-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.875703 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.875679    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96jzp\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-kube-api-access-96jzp\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.878281 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.878259    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:05.983783 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:05.983707    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:06.272327 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.272301    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"]
Nov 02 13:55:06.274323 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:06.274294    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c210602_64e7_4655_b4f6_102c28acca45.slice/crio-6c759cfc1bb5036bf6df8dcb0258dc53fb0ae50c6ac079887f3d372ea8f9bf8b WatchSource:0}: Error finding container 6c759cfc1bb5036bf6df8dcb0258dc53fb0ae50c6ac079887f3d372ea8f9bf8b: Status 404 returned error can't find the container with id 6c759cfc1bb5036bf6df8dcb0258dc53fb0ae50c6ac079887f3d372ea8f9bf8b
Nov 02 13:55:06.277359 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.277327    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:06.277460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.277409    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:06.277460 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.277450    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:06.654274 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.654243    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" event={"ID":"0c210602-64e7-4655-b4f6-102c28acca45","Type":"ContainerStarted","Data":"af1904c4b7ce1bf0ea416493235f825ce3a7c3470720ae0f0a2ebe26a350192d"}
Nov 02 13:55:06.654449 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.654294    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" event={"ID":"0c210602-64e7-4655-b4f6-102c28acca45","Type":"ContainerStarted","Data":"6c759cfc1bb5036bf6df8dcb0258dc53fb0ae50c6ac079887f3d372ea8f9bf8b"}
Nov 02 13:55:06.678468 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.678425    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podStartSLOduration=1.678412664 podStartE2EDuration="1.678412664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:06.677949543 +0000 UTC m=+683.256038416" watchObservedRunningTime="2026-11-02 13:55:06.678412664 +0000 UTC m=+683.256501531"
Nov 02 13:55:06.983857 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:06.983778    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:08.001592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:07.993682    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:55:08.001592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:07.993751    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:55:09.005591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:09.004664    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:55:09.005591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:09.004725    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:55:10.005945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:10.005912    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:55:10.006394 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:10.005973    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:55:10.985022 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:10.984986    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:55:10.985314 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:10.985285    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:55:11.671502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:11.671471    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:11.710957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:11.710929    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:11.722319 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:11.722296    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:55:11.990023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:11.989933    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:55:11.990280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:11.990252    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:55:12.984759 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:12.984722    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:55:12.985129 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:12.984781    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:55:13.988653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:13.988626    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:13.989093 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:13.988872    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:13.989526 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:13.989508    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 13:55:14.213239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:14.213209    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:14.226050 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:14.226028    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:14.732578 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:14.732528    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:55:21.431435 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:55:21.431393    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:55:35.009696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.009662    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"]
Nov 02 13:55:35.013546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.013524    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.017702 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.017680    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ee7df2-openshift-default-dockercfg-hc6nm\""
Nov 02 13:55:35.026413 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.026390    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"]
Nov 02 13:55:35.174962 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.174925    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmcg\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-kube-api-access-jnmcg\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175040    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175072    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175137    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a5f392-cfbd-4849-a885-6b13b0d84781-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175157    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175299 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175223    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175299 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175293    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175318    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.175378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.175353    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276642    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276702    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmcg\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-kube-api-access-jnmcg\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276747    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276770    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276786    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a5f392-cfbd-4849-a885-6b13b0d84781-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276804    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276865    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276892    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.276955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.276924    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.277264 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.277230    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.277323 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.277303    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.277445 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.277421    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.277513 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.277449    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.277774 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.277689    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a5f392-cfbd-4849-a885-6b13b0d84781-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.279244 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.279220    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.279350 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.279331    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.289864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.289843    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.290272 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.290250    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmcg\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-kube-api-access-jnmcg\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.344581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.334426    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:35.374529 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.374494    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"]
Nov 02 13:55:35.379389 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.379366    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.385516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.385494    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7a9443-openshift-default-dockercfg-m25kh\""
Nov 02 13:55:35.405983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.405944    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"]
Nov 02 13:55:35.478863 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.478821    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479024 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.478875    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479024 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.478944    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479024 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.479006    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.479125    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.479152    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.479199    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.479305    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.479374 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.479358    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zrz\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-kube-api-access-p2zrz\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.580817 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.580791    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.580968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.580831    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.580968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.580855    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrz\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-kube-api-access-p2zrz\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.580968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.580888    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.580968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.580923    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.580968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.580948    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.580987    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.581068    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.581100    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.581212    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581463 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.581322    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581523 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.581493    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581630 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.581524    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.581786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.581764    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.583483 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.583465    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.583597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.583580    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.592915 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.592890    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.593315 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.593298    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zrz\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-kube-api-access-p2zrz\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.650906 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.650882    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"]
Nov 02 13:55:35.652649 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:35.652627    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a5f392_cfbd_4849_a885_6b13b0d84781.slice/crio-f1494a46e67bc87811e6f235ba84c91329f941661d5a673c299f5e820b9bf040 WatchSource:0}: Error finding container f1494a46e67bc87811e6f235ba84c91329f941661d5a673c299f5e820b9bf040: Status 404 returned error can't find the container with id f1494a46e67bc87811e6f235ba84c91329f941661d5a673c299f5e820b9bf040
Nov 02 13:55:35.654865 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.654831    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:35.654976 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.654908    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:35.654976 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.654957    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:35.704375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.704353    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:35.859290 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.859197    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" event={"ID":"81a5f392-cfbd-4849-a885-6b13b0d84781","Type":"ContainerStarted","Data":"c9eea99107b0beb72c48d32618ad70e6bb1c9f76d084d2e62980e81773116a21"}
Nov 02 13:55:35.859290 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.859260    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" event={"ID":"81a5f392-cfbd-4849-a885-6b13b0d84781","Type":"ContainerStarted","Data":"f1494a46e67bc87811e6f235ba84c91329f941661d5a673c299f5e820b9bf040"}
Nov 02 13:55:35.883101 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.883051    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podStartSLOduration=1.883035775 podStartE2EDuration="1.883035775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:35.881914 +0000 UTC m=+712.460002869" watchObservedRunningTime="2026-11-02 13:55:35.883035775 +0000 UTC m=+712.461124645"
Nov 02 13:55:35.994106 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.994079    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"]
Nov 02 13:55:35.995800 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:35.995771    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651ba3ec_24ee_4e08_8c41_e77066d4dd96.slice/crio-cdbc9ffeac4d012bba6661e91f2cee46f1f31cf46a7926723f520de6e00c9518 WatchSource:0}: Error finding container cdbc9ffeac4d012bba6661e91f2cee46f1f31cf46a7926723f520de6e00c9518: Status 404 returned error can't find the container with id cdbc9ffeac4d012bba6661e91f2cee46f1f31cf46a7926723f520de6e00c9518
Nov 02 13:55:35.998886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.998857    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:35.998951 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.998920    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:35.998992 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:35.998950    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:55:36.336718 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:36.336686    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:36.337816 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:36.337797    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 13:55:36.337879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:36.337843    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 13:55:36.865971 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:36.865937    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" event={"ID":"651ba3ec-24ee-4e08-8c41-e77066d4dd96","Type":"ContainerStarted","Data":"71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b"}
Nov 02 13:55:36.866178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:36.865991    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" event={"ID":"651ba3ec-24ee-4e08-8c41-e77066d4dd96","Type":"ContainerStarted","Data":"cdbc9ffeac4d012bba6661e91f2cee46f1f31cf46a7926723f520de6e00c9518"}
Nov 02 13:55:36.906583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:36.906129    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podStartSLOduration=1.906093321 podStartE2EDuration="1.906093321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:36.900838555 +0000 UTC m=+713.478927422" watchObservedRunningTime="2026-11-02 13:55:36.906093321 +0000 UTC m=+713.484182189"
Nov 02 13:55:37.336951 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:37.336911    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 13:55:37.337347 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:37.336963    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 13:55:37.708641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:37.708529    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:37.713599 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:37.712663    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:55:37.713599 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:37.712733    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 13:55:38.345615 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:38.340053    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 13:55:38.345615 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:38.340126    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 13:55:38.713681 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:38.710059    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:55:38.713681 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:38.710119    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 13:55:39.336888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:39.336851    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 13:55:39.337052 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:39.336925    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 13:55:39.707644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:39.706927    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:55:39.707644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:39.706998    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 13:55:40.336973 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:40.336934    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 13:55:40.337127 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:40.336997    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 13:55:40.710667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:40.705932    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:55:40.710667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:40.705993    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 13:55:41.337026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:41.336984    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 13:55:41.337233 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:41.337042    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 13:55:41.709457 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:41.709364    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:55:41.709755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:41.709726    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 13:55:42.336407 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:42.336368    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 13:55:42.336842 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:42.336435    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 13:55:42.705878 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:42.705794    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:55:42.706074 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:42.705874    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 13:55:43.341305 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:43.341278    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:43.341729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:43.341595    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:43.342335 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:43.342320    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 13:55:43.709697 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:43.709632    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:43.710018 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:43.709999    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:43.710605 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:43.710589    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 13:55:48.198507 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.198470    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"]
Nov 02 13:55:48.203795 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.203778    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.208908 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.208887    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:55:48.209039 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.208974    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:55:48.211613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.211588    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"]
Nov 02 13:55:48.225462 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.225436    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.225659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.225638    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2lk\" (UniqueName: \"kubernetes.io/projected/9b66ac5a-c928-4404-9435-492ad0ca8ea6-kube-api-access-8j2lk\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.225716 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.225702    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.327320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.327274    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.327493 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.327391    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2lk\" (UniqueName: \"kubernetes.io/projected/9b66ac5a-c928-4404-9435-492ad0ca8ea6-kube-api-access-8j2lk\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.327493 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.327445    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.328149 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.328125    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.329734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.329715    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.340282 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.340259    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2lk\" (UniqueName: \"kubernetes.io/projected/9b66ac5a-c928-4404-9435-492ad0ca8ea6-kube-api-access-8j2lk\") pod \"maas-api-revoke-sub-df2720b9-2m5rf\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.519370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.519279    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:48.797188 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.797164    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"]
Nov 02 13:55:48.799266 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:48.799240    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b66ac5a_c928_4404_9435_492ad0ca8ea6.slice/crio-13095a0b582f64837ad483ba9b21714e68c65b3347d45ee6ce8143f1cbd214ed WatchSource:0}: Error finding container 13095a0b582f64837ad483ba9b21714e68c65b3347d45ee6ce8143f1cbd214ed: Status 404 returned error can't find the container with id 13095a0b582f64837ad483ba9b21714e68c65b3347d45ee6ce8143f1cbd214ed
Nov 02 13:55:48.801328 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:48.801313    5572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:55:49.011756 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:49.011724    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf" event={"ID":"9b66ac5a-c928-4404-9435-492ad0ca8ea6","Type":"ContainerStarted","Data":"13095a0b582f64837ad483ba9b21714e68c65b3347d45ee6ce8143f1cbd214ed"}
Nov 02 13:55:51.173699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.173663    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"]
Nov 02 13:55:51.178861 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.178835    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.186882 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.186851    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"]
Nov 02 13:55:51.261109 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.261074    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvmw\" (UniqueName: \"kubernetes.io/projected/227544da-1f67-49c9-9bd7-289b9328a0ba-kube-api-access-fbvmw\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.261310 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.261282    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.261381 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.261352    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.364116 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.364084    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvmw\" (UniqueName: \"kubernetes.io/projected/227544da-1f67-49c9-9bd7-289b9328a0ba-kube-api-access-fbvmw\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.364290 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.364206    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.364290 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.364233    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.365344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.365314    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.372005 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.371979    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.377913 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.377888    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvmw\" (UniqueName: \"kubernetes.io/projected/227544da-1f67-49c9-9bd7-289b9328a0ba-kube-api-access-fbvmw\") pod \"maas-api-revoke-sub-299ee5b8-dz7p4\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.435210 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:55:51.435115    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:55:51.498641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.498609    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:51.783080 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:51.783048    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"]
Nov 02 13:55:51.785378 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:51.785354    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227544da_1f67_49c9_9bd7_289b9328a0ba.slice/crio-1fb82d9c9d1c0fae1de08df52091d5ced5f55c7c89f6a6aee592300d9a1a3c4a WatchSource:0}: Error finding container 1fb82d9c9d1c0fae1de08df52091d5ced5f55c7c89f6a6aee592300d9a1a3c4a: Status 404 returned error can't find the container with id 1fb82d9c9d1c0fae1de08df52091d5ced5f55c7c89f6a6aee592300d9a1a3c4a
Nov 02 13:55:52.032712 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:52.032626    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4" event={"ID":"227544da-1f67-49c9-9bd7-289b9328a0ba","Type":"ContainerStarted","Data":"1fb82d9c9d1c0fae1de08df52091d5ced5f55c7c89f6a6aee592300d9a1a3c4a"}
Nov 02 13:55:53.045895 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:53.045817    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf" event={"ID":"9b66ac5a-c928-4404-9435-492ad0ca8ea6","Type":"ContainerStarted","Data":"1cb30395c80775332042afb0370a1a99cd6ff5b38c46c8aabe2233fe4be6b727"}
Nov 02 13:55:53.065406 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:53.065354    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf" podStartSLOduration=1.28986485 podStartE2EDuration="5.065338153s" totalImagesPullingTime="3.775473303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:53.063767824 +0000 UTC m=+729.641856692" watchObservedRunningTime="2026-11-02 13:55:53.065338153 +0000 UTC m=+729.643427019"
Nov 02 13:55:54.052399 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:54.052368    5572 generic.go:356] "Generic (PLEG): container finished" podID="9b66ac5a-c928-4404-9435-492ad0ca8ea6" containerID="1cb30395c80775332042afb0370a1a99cd6ff5b38c46c8aabe2233fe4be6b727" exitCode=0
Nov 02 13:55:54.052888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:54.052511    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf" event={"ID":"9b66ac5a-c928-4404-9435-492ad0ca8ea6","Type":"ContainerDied","Data":"1cb30395c80775332042afb0370a1a99cd6ff5b38c46c8aabe2233fe4be6b727"}
Nov 02 13:55:54.054107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:54.054089    5572 generic.go:356] "Generic (PLEG): container finished" podID="227544da-1f67-49c9-9bd7-289b9328a0ba" containerID="6cec2ba3200f535391b4d25583c9d4fabfd86e598553fb1393df5b7f609865e7" exitCode=0
Nov 02 13:55:54.054193 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:54.054141    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4" event={"ID":"227544da-1f67-49c9-9bd7-289b9328a0ba","Type":"ContainerDied","Data":"6cec2ba3200f535391b4d25583c9d4fabfd86e598553fb1393df5b7f609865e7"}
Nov 02 13:55:55.211910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.211883    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:55.241842 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.241819    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:55.315597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.315488    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-xks-ca\") pod \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") "
Nov 02 13:55:55.315597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.315535    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227544da-1f67-49c9-9bd7-289b9328a0ba-kube-api-access-fbvmw\" (UniqueName: \"kubernetes.io/projected/227544da-1f67-49c9-9bd7-289b9328a0ba-kube-api-access-fbvmw\") pod \"227544da-1f67-49c9-9bd7-289b9328a0ba\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") "
Nov 02 13:55:55.315828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.315611    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-openshift-ca\") pod \"227544da-1f67-49c9-9bd7-289b9328a0ba\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") "
Nov 02 13:55:55.315828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.315767    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b66ac5a-c928-4404-9435-492ad0ca8ea6-kube-api-access-8j2lk\" (UniqueName: \"kubernetes.io/projected/9b66ac5a-c928-4404-9435-492ad0ca8ea6-kube-api-access-8j2lk\") pod \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") "
Nov 02 13:55:55.315828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.315826    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-xks-ca\") pod \"227544da-1f67-49c9-9bd7-289b9328a0ba\" (UID: \"227544da-1f67-49c9-9bd7-289b9328a0ba\") "
Nov 02 13:55:55.315994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.315881    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-openshift-ca\") pod \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\" (UID: \"9b66ac5a-c928-4404-9435-492ad0ca8ea6\") "
Nov 02 13:55:55.315994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.315949    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-openshift-ca" pod "227544da-1f67-49c9-9bd7-289b9328a0ba" (UID: "227544da-1f67-49c9-9bd7-289b9328a0ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:55.316281 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.316251    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-openshift-ca" pod "9b66ac5a-c928-4404-9435-492ad0ca8ea6" (UID: "9b66ac5a-c928-4404-9435-492ad0ca8ea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:55.316583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.316496    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:55.316583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.316517    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:55.317952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.317928    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-xks-ca" pod "9b66ac5a-c928-4404-9435-492ad0ca8ea6" (UID: "9b66ac5a-c928-4404-9435-492ad0ca8ea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:55:55.317952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.317945    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227544da-1f67-49c9-9bd7-289b9328a0ba-kube-api-access-fbvmw" pod "227544da-1f67-49c9-9bd7-289b9328a0ba" (UID: "227544da-1f67-49c9-9bd7-289b9328a0ba"). InnerVolumeSpecName "kube-api-access-fbvmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:55.318082 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.317989    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-xks-ca" pod "227544da-1f67-49c9-9bd7-289b9328a0ba" (UID: "227544da-1f67-49c9-9bd7-289b9328a0ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:55:55.318082 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.318001    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b66ac5a-c928-4404-9435-492ad0ca8ea6-kube-api-access-8j2lk" pod "9b66ac5a-c928-4404-9435-492ad0ca8ea6" (UID: "9b66ac5a-c928-4404-9435-492ad0ca8ea6"). InnerVolumeSpecName "kube-api-access-8j2lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:55.417568 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.417532    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j2lk\" (UniqueName: \"kubernetes.io/projected/9b66ac5a-c928-4404-9435-492ad0ca8ea6-kube-api-access-8j2lk\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:55.417662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.417596    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227544da-1f67-49c9-9bd7-289b9328a0ba-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:55.417662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.417607    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b66ac5a-c928-4404-9435-492ad0ca8ea6-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:55.417662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.417615    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbvmw\" (UniqueName: \"kubernetes.io/projected/227544da-1f67-49c9-9bd7-289b9328a0ba-kube-api-access-fbvmw\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:55.476333 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.476304    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"]
Nov 02 13:55:55.476824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.476809    5572 cpu_manager.go:414] "containerMap: removing container" podUID="227544da-1f67-49c9-9bd7-289b9328a0ba" containerName="revoke-keys"
Nov 02 13:55:55.476871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.476828    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227544da-1f67-49c9-9bd7-289b9328a0ba" containerName="revoke-keys"
Nov 02 13:55:55.476871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.476864    5572 cpu_manager.go:414] "containerMap: removing container" podUID="9b66ac5a-c928-4404-9435-492ad0ca8ea6" containerName="revoke-keys"
Nov 02 13:55:55.476871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.476869    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b66ac5a-c928-4404-9435-492ad0ca8ea6" containerName="revoke-keys"
Nov 02 13:55:55.476981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.476972    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b66ac5a-c928-4404-9435-492ad0ca8ea6" containerName="revoke-keys"
Nov 02 13:55:55.477014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.476985    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="227544da-1f67-49c9-9bd7-289b9328a0ba" containerName="revoke-keys"
Nov 02 13:55:55.480326 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.480311    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.490894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.490868    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"]
Nov 02 13:55:55.518534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.518508    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbf4p\" (UniqueName: \"kubernetes.io/projected/051d5861-951e-49d9-bda4-0fe5929ced94-kube-api-access-bbf4p\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.518782 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.518756    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.518889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.518873    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.619625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.619530    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.619625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.619614    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.619820 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.619660    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbf4p\" (UniqueName: \"kubernetes.io/projected/051d5861-951e-49d9-bda4-0fe5929ced94-kube-api-access-bbf4p\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.620251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.620224    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.622109 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.622091    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.631920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.631900    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbf4p\" (UniqueName: \"kubernetes.io/projected/051d5861-951e-49d9-bda4-0fe5929ced94-kube-api-access-bbf4p\") pod \"maas-api-revoke-sub-37850f9c-xxdb6\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.797634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.797600    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:55.853345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.853315    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"]
Nov 02 13:55:55.860522 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.860498    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:55.869970 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.869879    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"]
Nov 02 13:55:55.925681 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.925646    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmctg\" (UniqueName: \"kubernetes.io/projected/79ce896e-9859-4665-8dc1-c330941aff19-kube-api-access-fmctg\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:55.925831 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.925734    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:55.925831 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:55.925816    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.026705 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.026671    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmctg\" (UniqueName: \"kubernetes.io/projected/79ce896e-9859-4665-8dc1-c330941aff19-kube-api-access-fmctg\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.026867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.026734    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.026867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.026774    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.027399 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.027377    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.029167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.029145    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.039191 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.039172    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmctg\" (UniqueName: \"kubernetes.io/projected/79ce896e-9859-4665-8dc1-c330941aff19-kube-api-access-fmctg\") pod \"maas-api-revoke-sub-d5c16d0d-xjpng\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.067090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.067073    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"
Nov 02 13:55:56.067184 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.067108    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4" event={"ID":"227544da-1f67-49c9-9bd7-289b9328a0ba","Type":"ContainerDied","Data":"1fb82d9c9d1c0fae1de08df52091d5ced5f55c7c89f6a6aee592300d9a1a3c4a"}
Nov 02 13:55:56.067184 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.067153    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fb82d9c9d1c0fae1de08df52091d5ced5f55c7c89f6a6aee592300d9a1a3c4a"
Nov 02 13:55:56.068385 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.068369    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"
Nov 02 13:55:56.068473 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.068385    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf" event={"ID":"9b66ac5a-c928-4404-9435-492ad0ca8ea6","Type":"ContainerDied","Data":"13095a0b582f64837ad483ba9b21714e68c65b3347d45ee6ce8143f1cbd214ed"}
Nov 02 13:55:56.068473 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.068437    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13095a0b582f64837ad483ba9b21714e68c65b3347d45ee6ce8143f1cbd214ed"
Nov 02 13:55:56.074724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.074702    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"]
Nov 02 13:55:56.075653 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:56.075630    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051d5861_951e_49d9_bda4_0fe5929ced94.slice/crio-03d6ac85fae8b6bb6cf95e275f0c0ff883f4e2c740373c36474cad26bde68294 WatchSource:0}: Error finding container 03d6ac85fae8b6bb6cf95e275f0c0ff883f4e2c740373c36474cad26bde68294: Status 404 returned error can't find the container with id 03d6ac85fae8b6bb6cf95e275f0c0ff883f4e2c740373c36474cad26bde68294
Nov 02 13:55:56.185418 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.185366    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:56.432344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.432317    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"]
Nov 02 13:55:56.436318 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.436264    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.449865 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.449834    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"]
Nov 02 13:55:56.474977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.474951    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"]
Nov 02 13:55:56.485324 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:56.485296    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ce896e_9859_4665_8dc1_c330941aff19.slice/crio-3f6b60fb53f868c84aff72ea2c3a67399b525d8d6f2bf1f64456ccb80bcd81b9 WatchSource:0}: Error finding container 3f6b60fb53f868c84aff72ea2c3a67399b525d8d6f2bf1f64456ccb80bcd81b9: Status 404 returned error can't find the container with id 3f6b60fb53f868c84aff72ea2c3a67399b525d8d6f2bf1f64456ccb80bcd81b9
Nov 02 13:55:56.532679 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.532654    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gdlv\" (UniqueName: \"kubernetes.io/projected/96e5a17a-a13c-43fd-825c-2d7a63c84570-kube-api-access-4gdlv\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.532777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.532709    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.532847 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.532826    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.637454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.634493    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gdlv\" (UniqueName: \"kubernetes.io/projected/96e5a17a-a13c-43fd-825c-2d7a63c84570-kube-api-access-4gdlv\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.637454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.634591    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.637454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.634699    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.637454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.635689    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.652454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.647379    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.663135 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.653191    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gdlv\" (UniqueName: \"kubernetes.io/projected/96e5a17a-a13c-43fd-825c-2d7a63c84570-kube-api-access-4gdlv\") pod \"maas-api-revoke-sub-367cec11-ghpfv\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:56.752058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:56.752030    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:57.034642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.034619    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"]
Nov 02 13:55:57.050625 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:55:57.050594    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e5a17a_a13c_43fd_825c_2d7a63c84570.slice/crio-48f99c5cc45dca4900f00b05335df9be88a1e873038ecd484406635e6e9c54f9 WatchSource:0}: Error finding container 48f99c5cc45dca4900f00b05335df9be88a1e873038ecd484406635e6e9c54f9: Status 404 returned error can't find the container with id 48f99c5cc45dca4900f00b05335df9be88a1e873038ecd484406635e6e9c54f9
Nov 02 13:55:57.075736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.075705    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv" event={"ID":"96e5a17a-a13c-43fd-825c-2d7a63c84570","Type":"ContainerStarted","Data":"48f99c5cc45dca4900f00b05335df9be88a1e873038ecd484406635e6e9c54f9"}
Nov 02 13:55:57.077084 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.077059    5572 generic.go:356] "Generic (PLEG): container finished" podID="051d5861-951e-49d9-bda4-0fe5929ced94" containerID="67a66b4eceef5c31b6dcab51155e99c6bdccadf2d265683eee1f5e2c4f25a5ae" exitCode=0
Nov 02 13:55:57.077185 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.077145    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6" event={"ID":"051d5861-951e-49d9-bda4-0fe5929ced94","Type":"ContainerDied","Data":"67a66b4eceef5c31b6dcab51155e99c6bdccadf2d265683eee1f5e2c4f25a5ae"}
Nov 02 13:55:57.077238 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.077190    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6" event={"ID":"051d5861-951e-49d9-bda4-0fe5929ced94","Type":"ContainerStarted","Data":"03d6ac85fae8b6bb6cf95e275f0c0ff883f4e2c740373c36474cad26bde68294"}
Nov 02 13:55:57.078537 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.078515    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng" event={"ID":"79ce896e-9859-4665-8dc1-c330941aff19","Type":"ContainerStarted","Data":"ce36cb3cb5ae0be452325f742298d44cd188b573218542c2f89a5b3480beeae7"}
Nov 02 13:55:57.078636 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.078553    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng" event={"ID":"79ce896e-9859-4665-8dc1-c330941aff19","Type":"ContainerStarted","Data":"3f6b60fb53f868c84aff72ea2c3a67399b525d8d6f2bf1f64456ccb80bcd81b9"}
Nov 02 13:55:57.116617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:57.116579    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng" podStartSLOduration=2.116545841 podStartE2EDuration="2.116545841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:57.115672367 +0000 UTC m=+733.693761235" watchObservedRunningTime="2026-11-02 13:55:57.116545841 +0000 UTC m=+733.694634708"
Nov 02 13:55:58.088218 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.088188    5572 generic.go:356] "Generic (PLEG): container finished" podID="79ce896e-9859-4665-8dc1-c330941aff19" containerID="ce36cb3cb5ae0be452325f742298d44cd188b573218542c2f89a5b3480beeae7" exitCode=0
Nov 02 13:55:58.088676 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.088327    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng" event={"ID":"79ce896e-9859-4665-8dc1-c330941aff19","Type":"ContainerDied","Data":"ce36cb3cb5ae0be452325f742298d44cd188b573218542c2f89a5b3480beeae7"}
Nov 02 13:55:58.090273 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.090251    5572 generic.go:356] "Generic (PLEG): container finished" podID="96e5a17a-a13c-43fd-825c-2d7a63c84570" containerID="ea8e2509ba6c27b28de48a4b355665bbbf67b11122292a43676500d6543c6502" exitCode=0
Nov 02 13:55:58.090400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.090350    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv" event={"ID":"96e5a17a-a13c-43fd-825c-2d7a63c84570","Type":"ContainerDied","Data":"ea8e2509ba6c27b28de48a4b355665bbbf67b11122292a43676500d6543c6502"}
Nov 02 13:55:58.310949 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.310920    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:58.410785 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.410000    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-openshift-ca\") pod \"051d5861-951e-49d9-bda4-0fe5929ced94\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") "
Nov 02 13:55:58.410785 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.410119    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-xks-ca\") pod \"051d5861-951e-49d9-bda4-0fe5929ced94\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") "
Nov 02 13:55:58.410785 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.410245    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051d5861-951e-49d9-bda4-0fe5929ced94-kube-api-access-bbf4p\" (UniqueName: \"kubernetes.io/projected/051d5861-951e-49d9-bda4-0fe5929ced94-kube-api-access-bbf4p\") pod \"051d5861-951e-49d9-bda4-0fe5929ced94\" (UID: \"051d5861-951e-49d9-bda4-0fe5929ced94\") "
Nov 02 13:55:58.412023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.411985    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-openshift-ca" pod "051d5861-951e-49d9-bda4-0fe5929ced94" (UID: "051d5861-951e-49d9-bda4-0fe5929ced94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:58.415361 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.415321    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-xks-ca" pod "051d5861-951e-49d9-bda4-0fe5929ced94" (UID: "051d5861-951e-49d9-bda4-0fe5929ced94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:55:58.415946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.415923    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051d5861-951e-49d9-bda4-0fe5929ced94-kube-api-access-bbf4p" pod "051d5861-951e-49d9-bda4-0fe5929ced94" (UID: "051d5861-951e-49d9-bda4-0fe5929ced94"). InnerVolumeSpecName "kube-api-access-bbf4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:58.512713 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.512678    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbf4p\" (UniqueName: \"kubernetes.io/projected/051d5861-951e-49d9-bda4-0fe5929ced94-kube-api-access-bbf4p\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:58.512948 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.512925    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:58.513089 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:58.513073    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051d5861-951e-49d9-bda4-0fe5929ced94-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:59.096940 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.096912    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"
Nov 02 13:55:59.097343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.096944    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6" event={"ID":"051d5861-951e-49d9-bda4-0fe5929ced94","Type":"ContainerDied","Data":"03d6ac85fae8b6bb6cf95e275f0c0ff883f4e2c740373c36474cad26bde68294"}
Nov 02 13:55:59.097343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.096987    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03d6ac85fae8b6bb6cf95e275f0c0ff883f4e2c740373c36474cad26bde68294"
Nov 02 13:55:59.234818 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.234789    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:55:59.251791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.251766    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:55:59.324068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.324030    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-openshift-ca\") pod \"79ce896e-9859-4665-8dc1-c330941aff19\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") "
Nov 02 13:55:59.324243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.324125    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ce896e-9859-4665-8dc1-c330941aff19-kube-api-access-fmctg\" (UniqueName: \"kubernetes.io/projected/79ce896e-9859-4665-8dc1-c330941aff19-kube-api-access-fmctg\") pod \"79ce896e-9859-4665-8dc1-c330941aff19\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") "
Nov 02 13:55:59.324313 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.324255    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-xks-ca\") pod \"79ce896e-9859-4665-8dc1-c330941aff19\" (UID: \"79ce896e-9859-4665-8dc1-c330941aff19\") "
Nov 02 13:55:59.324366 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.324347    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-openshift-ca" pod "79ce896e-9859-4665-8dc1-c330941aff19" (UID: "79ce896e-9859-4665-8dc1-c330941aff19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:59.324650 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.324630    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:59.326305 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.326276    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ce896e-9859-4665-8dc1-c330941aff19-kube-api-access-fmctg" pod "79ce896e-9859-4665-8dc1-c330941aff19" (UID: "79ce896e-9859-4665-8dc1-c330941aff19"). InnerVolumeSpecName "kube-api-access-fmctg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:59.326305 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.326294    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-xks-ca" pod "79ce896e-9859-4665-8dc1-c330941aff19" (UID: "79ce896e-9859-4665-8dc1-c330941aff19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:55:59.425065 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.424983    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e5a17a-a13c-43fd-825c-2d7a63c84570-kube-api-access-4gdlv\" (UniqueName: \"kubernetes.io/projected/96e5a17a-a13c-43fd-825c-2d7a63c84570-kube-api-access-4gdlv\") pod \"96e5a17a-a13c-43fd-825c-2d7a63c84570\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") "
Nov 02 13:55:59.425065 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.425067    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-openshift-ca\") pod \"96e5a17a-a13c-43fd-825c-2d7a63c84570\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") "
Nov 02 13:55:59.425303 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.425112    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-xks-ca\") pod \"96e5a17a-a13c-43fd-825c-2d7a63c84570\" (UID: \"96e5a17a-a13c-43fd-825c-2d7a63c84570\") "
Nov 02 13:55:59.425446 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.425422    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-openshift-ca" pod "96e5a17a-a13c-43fd-825c-2d7a63c84570" (UID: "96e5a17a-a13c-43fd-825c-2d7a63c84570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:59.425547 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.425531    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmctg\" (UniqueName: \"kubernetes.io/projected/79ce896e-9859-4665-8dc1-c330941aff19-kube-api-access-fmctg\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:59.425655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.425554    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ce896e-9859-4665-8dc1-c330941aff19-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:59.425655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.425591    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:59.427189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.427165    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-xks-ca" pod "96e5a17a-a13c-43fd-825c-2d7a63c84570" (UID: "96e5a17a-a13c-43fd-825c-2d7a63c84570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:55:59.427249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.427182    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e5a17a-a13c-43fd-825c-2d7a63c84570-kube-api-access-4gdlv" pod "96e5a17a-a13c-43fd-825c-2d7a63c84570" (UID: "96e5a17a-a13c-43fd-825c-2d7a63c84570"). InnerVolumeSpecName "kube-api-access-4gdlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:59.526806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.526772    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e5a17a-a13c-43fd-825c-2d7a63c84570-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:55:59.526806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:55:59.526802    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gdlv\" (UniqueName: \"kubernetes.io/projected/96e5a17a-a13c-43fd-825c-2d7a63c84570-kube-api-access-4gdlv\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:00.103250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:00.103218    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"
Nov 02 13:56:00.103720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:00.103250    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv" event={"ID":"96e5a17a-a13c-43fd-825c-2d7a63c84570","Type":"ContainerDied","Data":"48f99c5cc45dca4900f00b05335df9be88a1e873038ecd484406635e6e9c54f9"}
Nov 02 13:56:00.103720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:00.103343    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f99c5cc45dca4900f00b05335df9be88a1e873038ecd484406635e6e9c54f9"
Nov 02 13:56:00.104637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:00.104620    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"
Nov 02 13:56:00.104710 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:00.104674    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng" event={"ID":"79ce896e-9859-4665-8dc1-c330941aff19","Type":"ContainerDied","Data":"3f6b60fb53f868c84aff72ea2c3a67399b525d8d6f2bf1f64456ccb80bcd81b9"}
Nov 02 13:56:00.104746 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:00.104710    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6b60fb53f868c84aff72ea2c3a67399b525d8d6f2bf1f64456ccb80bcd81b9"
Nov 02 13:56:06.871971 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.871932    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"]
Nov 02 13:56:06.872482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872444    5572 cpu_manager.go:414] "containerMap: removing container" podUID="79ce896e-9859-4665-8dc1-c330941aff19" containerName="revoke-keys"
Nov 02 13:56:06.872482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872460    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ce896e-9859-4665-8dc1-c330941aff19" containerName="revoke-keys"
Nov 02 13:56:06.872482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872479    5572 cpu_manager.go:414] "containerMap: removing container" podUID="051d5861-951e-49d9-bda4-0fe5929ced94" containerName="revoke-keys"
Nov 02 13:56:06.872609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872484    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051d5861-951e-49d9-bda4-0fe5929ced94" containerName="revoke-keys"
Nov 02 13:56:06.872609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872497    5572 cpu_manager.go:414] "containerMap: removing container" podUID="96e5a17a-a13c-43fd-825c-2d7a63c84570" containerName="revoke-keys"
Nov 02 13:56:06.872609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872502    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e5a17a-a13c-43fd-825c-2d7a63c84570" containerName="revoke-keys"
Nov 02 13:56:06.872609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872607    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="051d5861-951e-49d9-bda4-0fe5929ced94" containerName="revoke-keys"
Nov 02 13:56:06.872728 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872619    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e5a17a-a13c-43fd-825c-2d7a63c84570" containerName="revoke-keys"
Nov 02 13:56:06.872728 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.872630    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ce896e-9859-4665-8dc1-c330941aff19" containerName="revoke-keys"
Nov 02 13:56:06.877971 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.877952    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:06.885806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.885780    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:06.889000 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.888977    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"]
Nov 02 13:56:06.907989 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.907960    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:56:06.944701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.944667    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:06.944894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.944818    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:06.945023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:06.944924    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwxn\" (UniqueName: \"kubernetes.io/projected/3c846bd9-df07-4ef4-a269-074f44291a34-kube-api-access-pkwxn\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.046137 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.046100    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.046334 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.046163    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwxn\" (UniqueName: \"kubernetes.io/projected/3c846bd9-df07-4ef4-a269-074f44291a34-kube-api-access-pkwxn\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.046334 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.046220    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.046954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.046931    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.048643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.048614    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.058781 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.058760    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwxn\" (UniqueName: \"kubernetes.io/projected/3c846bd9-df07-4ef4-a269-074f44291a34-kube-api-access-pkwxn\") pod \"maas-api-revoke-sub-09847548-ss4ch\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.193289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.193190    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:07.472257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.472233    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"]
Nov 02 13:56:07.475171 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:07.475126    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c846bd9_df07_4ef4_a269_074f44291a34.slice/crio-b3fcbdc5e29626db295733656296f9e93fbb18d5630956a17f41abb1b79069ed WatchSource:0}: Error finding container b3fcbdc5e29626db295733656296f9e93fbb18d5630956a17f41abb1b79069ed: Status 404 returned error can't find the container with id b3fcbdc5e29626db295733656296f9e93fbb18d5630956a17f41abb1b79069ed
Nov 02 13:56:07.817114 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.817080    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"]
Nov 02 13:56:07.821007 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.820984    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.830441 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.830413    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"]
Nov 02 13:56:07.854582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.854525    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.854582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.854579    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hqj\" (UniqueName: \"kubernetes.io/projected/6f0ec92c-8ed4-40cc-bed5-52759a164580-kube-api-access-d5hqj\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.854792 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.854705    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.955806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.955765    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.955806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.955807    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hqj\" (UniqueName: \"kubernetes.io/projected/6f0ec92c-8ed4-40cc-bed5-52759a164580-kube-api-access-d5hqj\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.956258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.955857    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.956485 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.956466    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.958255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.958235    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:07.968935 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:07.968908    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hqj\" (UniqueName: \"kubernetes.io/projected/6f0ec92c-8ed4-40cc-bed5-52759a164580-kube-api-access-d5hqj\") pod \"maas-api-revoke-sub-69fc5725-hgknv\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:08.136657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.136613    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:08.163602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.163554    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch" event={"ID":"3c846bd9-df07-4ef4-a269-074f44291a34","Type":"ContainerStarted","Data":"87ac09802ea30ff73a6b1651ac6df8fa86369fc309d65e0e19c510d9e64de4d3"}
Nov 02 13:56:08.163748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.163634    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch" event={"ID":"3c846bd9-df07-4ef4-a269-074f44291a34","Type":"ContainerStarted","Data":"b3fcbdc5e29626db295733656296f9e93fbb18d5630956a17f41abb1b79069ed"}
Nov 02 13:56:08.201434 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.197891    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch" podStartSLOduration=2.197872317 podStartE2EDuration="2.197872317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:08.193644516 +0000 UTC m=+744.771733386" watchObservedRunningTime="2026-11-02 13:56:08.197872317 +0000 UTC m=+744.775961185"
Nov 02 13:56:08.201434 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.198771    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"]
Nov 02 13:56:08.205726 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.205706    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.211603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.211575    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"]
Nov 02 13:56:08.264006 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.263950    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.264221 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.264048    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.264221 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.264115    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9zcl\" (UniqueName: \"kubernetes.io/projected/aa345a17-6c1f-427c-a14e-5609e7d1292b-kube-api-access-k9zcl\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.364922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.364874    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.365129 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.364953    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9zcl\" (UniqueName: \"kubernetes.io/projected/aa345a17-6c1f-427c-a14e-5609e7d1292b-kube-api-access-k9zcl\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.365205 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.365128    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.365760 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.365730    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.367743 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.367724    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.379247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.379220    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9zcl\" (UniqueName: \"kubernetes.io/projected/aa345a17-6c1f-427c-a14e-5609e7d1292b-kube-api-access-k9zcl\") pod \"maas-api-revoke-sub-fec63ed0-ksb5j\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.416881 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.416854    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"]
Nov 02 13:56:08.457662 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:08.457628    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0ec92c_8ed4_40cc_bed5_52759a164580.slice/crio-6fda66c735ebf869e808fc63a026f1abf892dd95a0213992ecd7472d4e2f6902 WatchSource:0}: Error finding container 6fda66c735ebf869e808fc63a026f1abf892dd95a0213992ecd7472d4e2f6902: Status 404 returned error can't find the container with id 6fda66c735ebf869e808fc63a026f1abf892dd95a0213992ecd7472d4e2f6902
Nov 02 13:56:08.528377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.528348    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:08.809471 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:08.809449    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"]
Nov 02 13:56:08.811960 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:08.811926    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa345a17_6c1f_427c_a14e_5609e7d1292b.slice/crio-49a759349f147314dcda207e466aa976c2d4f57cf973983aca7f6e5f6a40d4d3 WatchSource:0}: Error finding container 49a759349f147314dcda207e466aa976c2d4f57cf973983aca7f6e5f6a40d4d3: Status 404 returned error can't find the container with id 49a759349f147314dcda207e466aa976c2d4f57cf973983aca7f6e5f6a40d4d3
Nov 02 13:56:09.171208 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.171175    5572 generic.go:356] "Generic (PLEG): container finished" podID="3c846bd9-df07-4ef4-a269-074f44291a34" containerID="87ac09802ea30ff73a6b1651ac6df8fa86369fc309d65e0e19c510d9e64de4d3" exitCode=0
Nov 02 13:56:09.171651 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.171266    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch" event={"ID":"3c846bd9-df07-4ef4-a269-074f44291a34","Type":"ContainerDied","Data":"87ac09802ea30ff73a6b1651ac6df8fa86369fc309d65e0e19c510d9e64de4d3"}
Nov 02 13:56:09.173056 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.173026    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j" event={"ID":"aa345a17-6c1f-427c-a14e-5609e7d1292b","Type":"ContainerStarted","Data":"c027c665d57e5f0d30b2fc8307e3b12e44e577d7e10a806af7a8bf1952755321"}
Nov 02 13:56:09.173171 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.173080    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j" event={"ID":"aa345a17-6c1f-427c-a14e-5609e7d1292b","Type":"ContainerStarted","Data":"49a759349f147314dcda207e466aa976c2d4f57cf973983aca7f6e5f6a40d4d3"}
Nov 02 13:56:09.174595 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.174573    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv" event={"ID":"6f0ec92c-8ed4-40cc-bed5-52759a164580","Type":"ContainerStarted","Data":"c91a8dd257bc5ad8c6f7742f9d972a33ff477e3962dc00158d9dc77cdb7b10aa"}
Nov 02 13:56:09.174684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.174610    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv" event={"ID":"6f0ec92c-8ed4-40cc-bed5-52759a164580","Type":"ContainerStarted","Data":"6fda66c735ebf869e808fc63a026f1abf892dd95a0213992ecd7472d4e2f6902"}
Nov 02 13:56:09.209177 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.209127    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv" podStartSLOduration=2.209110418 podStartE2EDuration="2.209110418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:09.207791721 +0000 UTC m=+745.785880585" watchObservedRunningTime="2026-11-02 13:56:09.209110418 +0000 UTC m=+745.787199286"
Nov 02 13:56:09.228617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:09.228539    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j" podStartSLOduration=1.2285190049999999 podStartE2EDuration="1.228519005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:09.225537446 +0000 UTC m=+745.803626317" watchObservedRunningTime="2026-11-02 13:56:09.228519005 +0000 UTC m=+745.806607875"
Nov 02 13:56:10.181430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.181396    5572 generic.go:356] "Generic (PLEG): container finished" podID="6f0ec92c-8ed4-40cc-bed5-52759a164580" containerID="c91a8dd257bc5ad8c6f7742f9d972a33ff477e3962dc00158d9dc77cdb7b10aa" exitCode=0
Nov 02 13:56:10.181887 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.181479    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv" event={"ID":"6f0ec92c-8ed4-40cc-bed5-52759a164580","Type":"ContainerDied","Data":"c91a8dd257bc5ad8c6f7742f9d972a33ff477e3962dc00158d9dc77cdb7b10aa"}
Nov 02 13:56:10.183028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.183010    5572 generic.go:356] "Generic (PLEG): container finished" podID="aa345a17-6c1f-427c-a14e-5609e7d1292b" containerID="c027c665d57e5f0d30b2fc8307e3b12e44e577d7e10a806af7a8bf1952755321" exitCode=0
Nov 02 13:56:10.183086 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.183072    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j" event={"ID":"aa345a17-6c1f-427c-a14e-5609e7d1292b","Type":"ContainerDied","Data":"c027c665d57e5f0d30b2fc8307e3b12e44e577d7e10a806af7a8bf1952755321"}
Nov 02 13:56:10.366737 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.366687    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:10.405849 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.405820    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c846bd9-df07-4ef4-a269-074f44291a34-kube-api-access-pkwxn\" (UniqueName: \"kubernetes.io/projected/3c846bd9-df07-4ef4-a269-074f44291a34-kube-api-access-pkwxn\") pod \"3c846bd9-df07-4ef4-a269-074f44291a34\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") "
Nov 02 13:56:10.406041 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.405915    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-openshift-ca\") pod \"3c846bd9-df07-4ef4-a269-074f44291a34\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") "
Nov 02 13:56:10.406041 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.406002    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-xks-ca\") pod \"3c846bd9-df07-4ef4-a269-074f44291a34\" (UID: \"3c846bd9-df07-4ef4-a269-074f44291a34\") "
Nov 02 13:56:10.406316 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.406288    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-openshift-ca" pod "3c846bd9-df07-4ef4-a269-074f44291a34" (UID: "3c846bd9-df07-4ef4-a269-074f44291a34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:10.406436 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.406419    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:10.408344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.408313    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c846bd9-df07-4ef4-a269-074f44291a34-kube-api-access-pkwxn" pod "3c846bd9-df07-4ef4-a269-074f44291a34" (UID: "3c846bd9-df07-4ef4-a269-074f44291a34"). InnerVolumeSpecName "kube-api-access-pkwxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:10.408443 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.408356    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-xks-ca" pod "3c846bd9-df07-4ef4-a269-074f44291a34" (UID: "3c846bd9-df07-4ef4-a269-074f44291a34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:10.507501 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.507427    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c846bd9-df07-4ef4-a269-074f44291a34-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:10.507501 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.507453    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwxn\" (UniqueName: \"kubernetes.io/projected/3c846bd9-df07-4ef4-a269-074f44291a34-kube-api-access-pkwxn\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:10.925056 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.925022    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"]
Nov 02 13:56:10.925600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.925584    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3c846bd9-df07-4ef4-a269-074f44291a34" containerName="revoke-keys"
Nov 02 13:56:10.925642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.925603    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c846bd9-df07-4ef4-a269-074f44291a34" containerName="revoke-keys"
Nov 02 13:56:10.925917 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.925808    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c846bd9-df07-4ef4-a269-074f44291a34" containerName="revoke-keys"
Nov 02 13:56:10.929968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.929951    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:10.938123 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:10.938101    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"]
Nov 02 13:56:11.012367 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.012335    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxk56\" (UniqueName: \"kubernetes.io/projected/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-kube-api-access-jxk56\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.012541 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.012400    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.012541 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.012505    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.114124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.114092    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.114302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.114196    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxk56\" (UniqueName: \"kubernetes.io/projected/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-kube-api-access-jxk56\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.114302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.114266    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.114801 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.114778    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.116801 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.116785    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.129037 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.129010    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxk56\" (UniqueName: \"kubernetes.io/projected/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-kube-api-access-jxk56\") pod \"maas-api-revoke-sub-9e223e24-gsn46\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.189693 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.189618    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"
Nov 02 13:56:11.189693 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.189629    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch" event={"ID":"3c846bd9-df07-4ef4-a269-074f44291a34","Type":"ContainerDied","Data":"b3fcbdc5e29626db295733656296f9e93fbb18d5630956a17f41abb1b79069ed"}
Nov 02 13:56:11.190145 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.189693    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3fcbdc5e29626db295733656296f9e93fbb18d5630956a17f41abb1b79069ed"
Nov 02 13:56:11.245757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.245727    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:11.351844 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.351819    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:11.399325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.399304    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:11.418087 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.418066    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-openshift-ca\") pod \"aa345a17-6c1f-427c-a14e-5609e7d1292b\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") "
Nov 02 13:56:11.418192 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.418122    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa345a17-6c1f-427c-a14e-5609e7d1292b-kube-api-access-k9zcl\" (UniqueName: \"kubernetes.io/projected/aa345a17-6c1f-427c-a14e-5609e7d1292b-kube-api-access-k9zcl\") pod \"aa345a17-6c1f-427c-a14e-5609e7d1292b\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") "
Nov 02 13:56:11.418240 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.418198    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-xks-ca\") pod \"aa345a17-6c1f-427c-a14e-5609e7d1292b\" (UID: \"aa345a17-6c1f-427c-a14e-5609e7d1292b\") "
Nov 02 13:56:11.418392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.418371    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-openshift-ca" pod "aa345a17-6c1f-427c-a14e-5609e7d1292b" (UID: "aa345a17-6c1f-427c-a14e-5609e7d1292b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:11.421521 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.421448    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa345a17-6c1f-427c-a14e-5609e7d1292b-kube-api-access-k9zcl" pod "aa345a17-6c1f-427c-a14e-5609e7d1292b" (UID: "aa345a17-6c1f-427c-a14e-5609e7d1292b"). InnerVolumeSpecName "kube-api-access-k9zcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:11.422082 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.421964    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-xks-ca" pod "aa345a17-6c1f-427c-a14e-5609e7d1292b" (UID: "aa345a17-6c1f-427c-a14e-5609e7d1292b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:11.428183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.422219    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:11.428183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.422251    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa345a17-6c1f-427c-a14e-5609e7d1292b-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:11.428183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.422266    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9zcl\" (UniqueName: \"kubernetes.io/projected/aa345a17-6c1f-427c-a14e-5609e7d1292b-kube-api-access-k9zcl\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:11.523476 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.523400    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-xks-ca\") pod \"6f0ec92c-8ed4-40cc-bed5-52759a164580\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") "
Nov 02 13:56:11.523476 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.523467    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0ec92c-8ed4-40cc-bed5-52759a164580-kube-api-access-d5hqj\" (UniqueName: \"kubernetes.io/projected/6f0ec92c-8ed4-40cc-bed5-52759a164580-kube-api-access-d5hqj\") pod \"6f0ec92c-8ed4-40cc-bed5-52759a164580\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") "
Nov 02 13:56:11.523675 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.523530    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-openshift-ca\") pod \"6f0ec92c-8ed4-40cc-bed5-52759a164580\" (UID: \"6f0ec92c-8ed4-40cc-bed5-52759a164580\") "
Nov 02 13:56:11.523830 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.523810    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-openshift-ca" pod "6f0ec92c-8ed4-40cc-bed5-52759a164580" (UID: "6f0ec92c-8ed4-40cc-bed5-52759a164580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:11.524079 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.524059    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:11.525649 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.525630    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0ec92c-8ed4-40cc-bed5-52759a164580-kube-api-access-d5hqj" pod "6f0ec92c-8ed4-40cc-bed5-52759a164580" (UID: "6f0ec92c-8ed4-40cc-bed5-52759a164580"). InnerVolumeSpecName "kube-api-access-d5hqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:11.525719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.525687    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-xks-ca" pod "6f0ec92c-8ed4-40cc-bed5-52759a164580" (UID: "6f0ec92c-8ed4-40cc-bed5-52759a164580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:11.538969 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:11.538945    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5d7ce5_bf2f_4036_a632_82baf1f0b9d7.slice/crio-9d3f50967d4652664a504f25512f10b69a40e46d972432195be49d4febf76e98 WatchSource:0}: Error finding container 9d3f50967d4652664a504f25512f10b69a40e46d972432195be49d4febf76e98: Status 404 returned error can't find the container with id 9d3f50967d4652664a504f25512f10b69a40e46d972432195be49d4febf76e98
Nov 02 13:56:11.543429 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.543408    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"]
Nov 02 13:56:11.625409 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.625381    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0ec92c-8ed4-40cc-bed5-52759a164580-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:11.625409 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:11.625408    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5hqj\" (UniqueName: \"kubernetes.io/projected/6f0ec92c-8ed4-40cc-bed5-52759a164580-kube-api-access-d5hqj\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:12.196637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.196609    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"
Nov 02 13:56:12.197076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.196610    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j" event={"ID":"aa345a17-6c1f-427c-a14e-5609e7d1292b","Type":"ContainerDied","Data":"49a759349f147314dcda207e466aa976c2d4f57cf973983aca7f6e5f6a40d4d3"}
Nov 02 13:56:12.197076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.196737    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a759349f147314dcda207e466aa976c2d4f57cf973983aca7f6e5f6a40d4d3"
Nov 02 13:56:12.198075 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.198051    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"
Nov 02 13:56:12.198215 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.198088    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv" event={"ID":"6f0ec92c-8ed4-40cc-bed5-52759a164580","Type":"ContainerDied","Data":"6fda66c735ebf869e808fc63a026f1abf892dd95a0213992ecd7472d4e2f6902"}
Nov 02 13:56:12.198215 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.198126    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fda66c735ebf869e808fc63a026f1abf892dd95a0213992ecd7472d4e2f6902"
Nov 02 13:56:12.199840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.199815    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46" event={"ID":"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7","Type":"ContainerStarted","Data":"923efe050c74fe4b671065a832d059bf9045356174b50dbd4ddddba9c45234f0"}
Nov 02 13:56:12.199966 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.199865    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46" event={"ID":"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7","Type":"ContainerStarted","Data":"9d3f50967d4652664a504f25512f10b69a40e46d972432195be49d4febf76e98"}
Nov 02 13:56:12.226211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.226167    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46" podStartSLOduration=2.22615598 podStartE2EDuration="2.22615598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:12.2239432 +0000 UTC m=+748.802032067" watchObservedRunningTime="2026-11-02 13:56:12.22615598 +0000 UTC m=+748.804244872"
Nov 02 13:56:12.323122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.323091    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"]
Nov 02 13:56:12.323687 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.323670    5572 cpu_manager.go:414] "containerMap: removing container" podUID="6f0ec92c-8ed4-40cc-bed5-52759a164580" containerName="revoke-keys"
Nov 02 13:56:12.323777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.323688    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0ec92c-8ed4-40cc-bed5-52759a164580" containerName="revoke-keys"
Nov 02 13:56:12.323777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.323706    5572 cpu_manager.go:414] "containerMap: removing container" podUID="aa345a17-6c1f-427c-a14e-5609e7d1292b" containerName="revoke-keys"
Nov 02 13:56:12.323777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.323713    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa345a17-6c1f-427c-a14e-5609e7d1292b" containerName="revoke-keys"
Nov 02 13:56:12.323945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.323854    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0ec92c-8ed4-40cc-bed5-52759a164580" containerName="revoke-keys"
Nov 02 13:56:12.323945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.323869    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa345a17-6c1f-427c-a14e-5609e7d1292b" containerName="revoke-keys"
Nov 02 13:56:12.327192 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.327173    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.336692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.336670    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"]
Nov 02 13:56:12.434684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.434652    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.434817 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.434778    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.434859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.434851    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl44j\" (UniqueName: \"kubernetes.io/projected/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-kube-api-access-kl44j\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.536354 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.536326    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.536524 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.536404    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl44j\" (UniqueName: \"kubernetes.io/projected/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-kube-api-access-kl44j\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.536524 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.536463    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.537059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.537041    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.538796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.538779    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.549237 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.549211    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl44j\" (UniqueName: \"kubernetes.io/projected/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-kube-api-access-kl44j\") pod \"maas-api-revoke-sub-30cc84a7-k4pxd\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.641232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.641204    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:12.920005 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:12.919979    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"]
Nov 02 13:56:12.921465 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:12.921439    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2c6ae6_0e92_4781_85c3_6e0b1ad40c00.slice/crio-a30590fc1cd42573407ec1e101dbeaef45ed255036369ada3d72c22b1626bfa2 WatchSource:0}: Error finding container a30590fc1cd42573407ec1e101dbeaef45ed255036369ada3d72c22b1626bfa2: Status 404 returned error can't find the container with id a30590fc1cd42573407ec1e101dbeaef45ed255036369ada3d72c22b1626bfa2
Nov 02 13:56:13.206743 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:13.206716    5572 generic.go:356] "Generic (PLEG): container finished" podID="bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" containerID="923efe050c74fe4b671065a832d059bf9045356174b50dbd4ddddba9c45234f0" exitCode=0
Nov 02 13:56:13.207117 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:13.206799    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46" event={"ID":"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7","Type":"ContainerDied","Data":"923efe050c74fe4b671065a832d059bf9045356174b50dbd4ddddba9c45234f0"}
Nov 02 13:56:13.208175 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:13.208151    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd" event={"ID":"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00","Type":"ContainerStarted","Data":"a30590fc1cd42573407ec1e101dbeaef45ed255036369ada3d72c22b1626bfa2"}
Nov 02 13:56:13.941591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:13.940152    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"]
Nov 02 13:56:13.965876 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:13.964303    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"]
Nov 02 13:56:13.965876 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:13.964427    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.058943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.058914    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58223644-0663-4482-99a6-18baa723492b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.059112 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.058981    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58223644-0663-4482-99a6-18baa723492b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.059156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.059110    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6zh\" (UniqueName: \"kubernetes.io/projected/58223644-0663-4482-99a6-18baa723492b-kube-api-access-8k6zh\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.160746 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.160663    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58223644-0663-4482-99a6-18baa723492b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.160746 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.160723    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58223644-0663-4482-99a6-18baa723492b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.160960 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.160801    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6zh\" (UniqueName: \"kubernetes.io/projected/58223644-0663-4482-99a6-18baa723492b-kube-api-access-8k6zh\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.161396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.161376    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58223644-0663-4482-99a6-18baa723492b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.163262 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.163240    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58223644-0663-4482-99a6-18baa723492b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.173437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.173416    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6zh\" (UniqueName: \"kubernetes.io/projected/58223644-0663-4482-99a6-18baa723492b-kube-api-access-8k6zh\") pod \"maas-api-revoke-sub-0fffa010-q6v4p\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.218828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.218797    5572 generic.go:356] "Generic (PLEG): container finished" podID="de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" containerID="52d50a850a2d624b8e956c7f3bfaf11c2b67b0b020e4b84c53cdf7014a0dd5f2" exitCode=0
Nov 02 13:56:14.219207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.218886    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd" event={"ID":"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00","Type":"ContainerDied","Data":"52d50a850a2d624b8e956c7f3bfaf11c2b67b0b020e4b84c53cdf7014a0dd5f2"}
Nov 02 13:56:14.283952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.283907    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:14.331729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.331701    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"]
Nov 02 13:56:14.345424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.345397    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.352012 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.351941    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"]
Nov 02 13:56:14.380952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.380930    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:14.464047 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.463972    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-openshift-ca\") pod \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") "
Nov 02 13:56:14.464192 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.464058    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-kube-api-access-jxk56\" (UniqueName: \"kubernetes.io/projected/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-kube-api-access-jxk56\") pod \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") "
Nov 02 13:56:14.464258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.464190    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-xks-ca\") pod \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\" (UID: \"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7\") "
Nov 02 13:56:14.464313 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.464284    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-openshift-ca" pod "bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" (UID: "bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:14.464450 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.464426    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbzh\" (UniqueName: \"kubernetes.io/projected/dc5e08c1-af22-4809-860a-73b3a152d66a-kube-api-access-rjbzh\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.464805 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.464650    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.464888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.464871    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.465128 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.465112    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:14.466264 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.466241    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-xks-ca" pod "bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" (UID: "bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:14.466325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.466296    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-kube-api-access-jxk56" pod "bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" (UID: "bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7"). InnerVolumeSpecName "kube-api-access-jxk56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:14.566171 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.566137    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbzh\" (UniqueName: \"kubernetes.io/projected/dc5e08c1-af22-4809-860a-73b3a152d66a-kube-api-access-rjbzh\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.566322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.566200    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.566322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.566231    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.566322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.566300    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxk56\" (UniqueName: \"kubernetes.io/projected/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-kube-api-access-jxk56\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:14.566322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.566312    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:14.566843 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.566819    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.568714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.568697    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.578487 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.578466    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbzh\" (UniqueName: \"kubernetes.io/projected/dc5e08c1-af22-4809-860a-73b3a152d66a-kube-api-access-rjbzh\") pod \"maas-api-revoke-sub-1566c6af-zg69h\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:14.579643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.579625    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"]
Nov 02 13:56:14.581277 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:14.581256    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58223644_0663_4482_99a6_18baa723492b.slice/crio-afd96101949968c064a842ac0d28550b30d93777a8084961ab4acb599fb4f1eb WatchSource:0}: Error finding container afd96101949968c064a842ac0d28550b30d93777a8084961ab4acb599fb4f1eb: Status 404 returned error can't find the container with id afd96101949968c064a842ac0d28550b30d93777a8084961ab4acb599fb4f1eb
Nov 02 13:56:14.692930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:14.692905    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:15.024179 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.024023    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"]
Nov 02 13:56:15.036941 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:15.036287    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5e08c1_af22_4809_860a_73b3a152d66a.slice/crio-16fccdb18eacad6d2a699435469b034cecfa646523a675d77df4f5f080a0a71c WatchSource:0}: Error finding container 16fccdb18eacad6d2a699435469b034cecfa646523a675d77df4f5f080a0a71c: Status 404 returned error can't find the container with id 16fccdb18eacad6d2a699435469b034cecfa646523a675d77df4f5f080a0a71c
Nov 02 13:56:15.228584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.228429    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h" event={"ID":"dc5e08c1-af22-4809-860a-73b3a152d66a","Type":"ContainerStarted","Data":"16fccdb18eacad6d2a699435469b034cecfa646523a675d77df4f5f080a0a71c"}
Nov 02 13:56:15.233583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.229983    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46" event={"ID":"bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7","Type":"ContainerDied","Data":"9d3f50967d4652664a504f25512f10b69a40e46d972432195be49d4febf76e98"}
Nov 02 13:56:15.233583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.230032    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d3f50967d4652664a504f25512f10b69a40e46d972432195be49d4febf76e98"
Nov 02 13:56:15.233583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.230104    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"
Nov 02 13:56:15.248551 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.236524    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p" event={"ID":"58223644-0663-4482-99a6-18baa723492b","Type":"ContainerStarted","Data":"55b2533454401fe19cb30781842b2f686eddf8d03a1f509ffebd91db5b0dd15f"}
Nov 02 13:56:15.248551 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.236735    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p" event={"ID":"58223644-0663-4482-99a6-18baa723492b","Type":"ContainerStarted","Data":"afd96101949968c064a842ac0d28550b30d93777a8084961ab4acb599fb4f1eb"}
Nov 02 13:56:15.288643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.284600    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p" podStartSLOduration=2.284580065 podStartE2EDuration="2.284580065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:15.266163192 +0000 UTC m=+751.844252060" watchObservedRunningTime="2026-11-02 13:56:15.284580065 +0000 UTC m=+751.862668929"
Nov 02 13:56:15.926587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.894868    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:15.926587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.913207    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-openshift-ca\") pod \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") "
Nov 02 13:56:15.926587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.913273    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-xks-ca\") pod \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") "
Nov 02 13:56:15.926587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.913336    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-kube-api-access-kl44j\" (UniqueName: \"kubernetes.io/projected/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-kube-api-access-kl44j\") pod \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\" (UID: \"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00\") "
Nov 02 13:56:15.960598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.960340    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-openshift-ca" pod "de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" (UID: "de2c6ae6-0e92-4781-85c3-6e0b1ad40c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:15.964590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:15.963438    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-xks-ca" pod "de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" (UID: "de2c6ae6-0e92-4781-85c3-6e0b1ad40c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:16.039614 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.038294    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:16.039614 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.038336    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:16.125860 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.125768    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-kube-api-access-kl44j" pod "de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" (UID: "de2c6ae6-0e92-4781-85c3-6e0b1ad40c00"). InnerVolumeSpecName "kube-api-access-kl44j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:16.146592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.139396    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl44j\" (UniqueName: \"kubernetes.io/projected/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00-kube-api-access-kl44j\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:16.374585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.373604    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h" event={"ID":"dc5e08c1-af22-4809-860a-73b3a152d66a","Type":"ContainerStarted","Data":"0b7737786343130063824addcff5cd308e2a0badcc9f2dc214d264a9166cf9eb"}
Nov 02 13:56:16.399647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.387319    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd" event={"ID":"de2c6ae6-0e92-4781-85c3-6e0b1ad40c00","Type":"ContainerDied","Data":"a30590fc1cd42573407ec1e101dbeaef45ed255036369ada3d72c22b1626bfa2"}
Nov 02 13:56:16.399647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.387386    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a30590fc1cd42573407ec1e101dbeaef45ed255036369ada3d72c22b1626bfa2"
Nov 02 13:56:16.399647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.387467    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"
Nov 02 13:56:16.506945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.506701    5572 generic.go:356] "Generic (PLEG): container finished" podID="58223644-0663-4482-99a6-18baa723492b" containerID="55b2533454401fe19cb30781842b2f686eddf8d03a1f509ffebd91db5b0dd15f" exitCode=0
Nov 02 13:56:16.506945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.506775    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p" event={"ID":"58223644-0663-4482-99a6-18baa723492b","Type":"ContainerDied","Data":"55b2533454401fe19cb30781842b2f686eddf8d03a1f509ffebd91db5b0dd15f"}
Nov 02 13:56:16.550587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:16.545534    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h" podStartSLOduration=2.545494268 podStartE2EDuration="2.545494268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:16.424169082 +0000 UTC m=+753.002257951" watchObservedRunningTime="2026-11-02 13:56:16.545494268 +0000 UTC m=+753.123583135"
Nov 02 13:56:16.688633 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:56:16.688503    5572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5e08c1_af22_4809_860a_73b3a152d66a.slice/crio-0b7737786343130063824addcff5cd308e2a0badcc9f2dc214d264a9166cf9eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5e08c1_af22_4809_860a_73b3a152d66a.slice/crio-conmon-0b7737786343130063824addcff5cd308e2a0badcc9f2dc214d264a9166cf9eb.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:56:17.547594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:17.538087    5572 generic.go:356] "Generic (PLEG): container finished" podID="dc5e08c1-af22-4809-860a-73b3a152d66a" containerID="0b7737786343130063824addcff5cd308e2a0badcc9f2dc214d264a9166cf9eb" exitCode=0
Nov 02 13:56:17.552105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:17.548195    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h" event={"ID":"dc5e08c1-af22-4809-860a-73b3a152d66a","Type":"ContainerDied","Data":"0b7737786343130063824addcff5cd308e2a0badcc9f2dc214d264a9166cf9eb"}
Nov 02 13:56:18.072999 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.072965    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:18.173100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.172093    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58223644-0663-4482-99a6-18baa723492b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58223644-0663-4482-99a6-18baa723492b-maas-api-xks-ca\") pod \"58223644-0663-4482-99a6-18baa723492b\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") "
Nov 02 13:56:18.173100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.172224    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58223644-0663-4482-99a6-18baa723492b-kube-api-access-8k6zh\" (UniqueName: \"kubernetes.io/projected/58223644-0663-4482-99a6-18baa723492b-kube-api-access-8k6zh\") pod \"58223644-0663-4482-99a6-18baa723492b\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") "
Nov 02 13:56:18.173100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.172362    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58223644-0663-4482-99a6-18baa723492b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58223644-0663-4482-99a6-18baa723492b-maas-api-openshift-ca\") pod \"58223644-0663-4482-99a6-18baa723492b\" (UID: \"58223644-0663-4482-99a6-18baa723492b\") "
Nov 02 13:56:18.173916 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.173600    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58223644-0663-4482-99a6-18baa723492b-maas-api-openshift-ca" pod "58223644-0663-4482-99a6-18baa723492b" (UID: "58223644-0663-4482-99a6-18baa723492b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:18.228282 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.228193    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58223644-0663-4482-99a6-18baa723492b-kube-api-access-8k6zh" pod "58223644-0663-4482-99a6-18baa723492b" (UID: "58223644-0663-4482-99a6-18baa723492b"). InnerVolumeSpecName "kube-api-access-8k6zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:18.231116 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.231063    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58223644-0663-4482-99a6-18baa723492b-maas-api-xks-ca" pod "58223644-0663-4482-99a6-18baa723492b" (UID: "58223644-0663-4482-99a6-18baa723492b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:18.273627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.273509    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58223644-0663-4482-99a6-18baa723492b-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:18.273627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.273549    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58223644-0663-4482-99a6-18baa723492b-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:18.273627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.273588    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k6zh\" (UniqueName: \"kubernetes.io/projected/58223644-0663-4482-99a6-18baa723492b-kube-api-access-8k6zh\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:18.563459 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.563427    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"
Nov 02 13:56:18.564018 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.563457    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p" event={"ID":"58223644-0663-4482-99a6-18baa723492b","Type":"ContainerDied","Data":"afd96101949968c064a842ac0d28550b30d93777a8084961ab4acb599fb4f1eb"}
Nov 02 13:56:18.564018 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.563511    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd96101949968c064a842ac0d28550b30d93777a8084961ab4acb599fb4f1eb"
Nov 02 13:56:18.795459 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.795435    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:18.886181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.886097    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-xks-ca\") pod \"dc5e08c1-af22-4809-860a-73b3a152d66a\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") "
Nov 02 13:56:18.886181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.886165    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-openshift-ca\") pod \"dc5e08c1-af22-4809-860a-73b3a152d66a\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") "
Nov 02 13:56:18.886430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.886261    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5e08c1-af22-4809-860a-73b3a152d66a-kube-api-access-rjbzh\" (UniqueName: \"kubernetes.io/projected/dc5e08c1-af22-4809-860a-73b3a152d66a-kube-api-access-rjbzh\") pod \"dc5e08c1-af22-4809-860a-73b3a152d66a\" (UID: \"dc5e08c1-af22-4809-860a-73b3a152d66a\") "
Nov 02 13:56:18.886546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.886521    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-openshift-ca" pod "dc5e08c1-af22-4809-860a-73b3a152d66a" (UID: "dc5e08c1-af22-4809-860a-73b3a152d66a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:18.886805 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.886785    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:18.888512 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.888487    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-xks-ca" pod "dc5e08c1-af22-4809-860a-73b3a152d66a" (UID: "dc5e08c1-af22-4809-860a-73b3a152d66a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:18.888708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.888690    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5e08c1-af22-4809-860a-73b3a152d66a-kube-api-access-rjbzh" pod "dc5e08c1-af22-4809-860a-73b3a152d66a" (UID: "dc5e08c1-af22-4809-860a-73b3a152d66a"). InnerVolumeSpecName "kube-api-access-rjbzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:18.988047 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.988015    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbzh\" (UniqueName: \"kubernetes.io/projected/dc5e08c1-af22-4809-860a-73b3a152d66a-kube-api-access-rjbzh\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:18.988047 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:18.988044    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5e08c1-af22-4809-860a-73b3a152d66a-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:19.161593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.161489    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"]
Nov 02 13:56:19.162101 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162081    5572 cpu_manager.go:414] "containerMap: removing container" podUID="dc5e08c1-af22-4809-860a-73b3a152d66a" containerName="revoke-keys"
Nov 02 13:56:19.162207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162104    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5e08c1-af22-4809-860a-73b3a152d66a" containerName="revoke-keys"
Nov 02 13:56:19.162207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162144    5572 cpu_manager.go:414] "containerMap: removing container" podUID="bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" containerName="revoke-keys"
Nov 02 13:56:19.162207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162153    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" containerName="revoke-keys"
Nov 02 13:56:19.162207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162188    5572 cpu_manager.go:414] "containerMap: removing container" podUID="de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" containerName="revoke-keys"
Nov 02 13:56:19.162207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162195    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" containerName="revoke-keys"
Nov 02 13:56:19.162496 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162235    5572 cpu_manager.go:414] "containerMap: removing container" podUID="58223644-0663-4482-99a6-18baa723492b" containerName="revoke-keys"
Nov 02 13:56:19.162496 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162244    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58223644-0663-4482-99a6-18baa723492b" containerName="revoke-keys"
Nov 02 13:56:19.162496 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162413    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="58223644-0663-4482-99a6-18baa723492b" containerName="revoke-keys"
Nov 02 13:56:19.162496 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162430    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5e08c1-af22-4809-860a-73b3a152d66a" containerName="revoke-keys"
Nov 02 13:56:19.162496 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162443    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" containerName="revoke-keys"
Nov 02 13:56:19.162496 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.162458    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" containerName="revoke-keys"
Nov 02 13:56:19.166068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.166046    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.179750 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.179715    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"]
Nov 02 13:56:19.180374 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.180353    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ee7df2\"/\"openshift-service-ca.crt\""
Nov 02 13:56:19.180503 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.180490    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ee7df2\"/\"distinct-w2-ee7df2-kserve-self-signed-certs\""
Nov 02 13:56:19.180614 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.180602    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ee7df2\"/\"default-dockercfg-pjdxg\""
Nov 02 13:56:19.180672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.180650    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ee7df2\"/\"kube-root-ca.crt\""
Nov 02 13:56:19.291741 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.291703    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9mcm\" (UniqueName: \"kubernetes.io/projected/59e91d94-7c4f-4d9e-ac04-04745b267604-kube-api-access-v9mcm\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.291924 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.291761    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-tmp-dir\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.291924 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.291846    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-model-cache\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.292012 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.291951    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-dshm\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.292050 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.292017    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e91d94-7c4f-4d9e-ac04-04745b267604-tls-certs\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.292081 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.292055    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-home\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.395894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.395855    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-dshm\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.395963    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e91d94-7c4f-4d9e-ac04-04745b267604-tls-certs\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.395990    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-home\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396208 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.396136    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9mcm\" (UniqueName: \"kubernetes.io/projected/59e91d94-7c4f-4d9e-ac04-04745b267604-kube-api-access-v9mcm\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396208 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.396192    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-tmp-dir\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.396218    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-model-cache\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396465 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.396434    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-home\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.396615    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-model-cache\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.396697 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.396675    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-tmp-dir\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.398397 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.398379    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-dshm\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.399113 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.399082    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e91d94-7c4f-4d9e-ac04-04745b267604-tls-certs\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.409401 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.409039    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9mcm\" (UniqueName: \"kubernetes.io/projected/59e91d94-7c4f-4d9e-ac04-04745b267604-kube-api-access-v9mcm\") pod \"distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.483612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.483502    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:19.571646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.571548    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"
Nov 02 13:56:19.572023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.571606    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h" event={"ID":"dc5e08c1-af22-4809-860a-73b3a152d66a","Type":"ContainerDied","Data":"16fccdb18eacad6d2a699435469b034cecfa646523a675d77df4f5f080a0a71c"}
Nov 02 13:56:19.572023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.571745    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16fccdb18eacad6d2a699435469b034cecfa646523a675d77df4f5f080a0a71c"
Nov 02 13:56:19.774774 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:19.774750    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"]
Nov 02 13:56:19.776189 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:19.776164    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59e91d94_7c4f_4d9e_ac04_04745b267604.slice/crio-058e7c9af90ce1310f6268a5f053e545a07369d13ddd9fe508130c74ba601814 WatchSource:0}: Error finding container 058e7c9af90ce1310f6268a5f053e545a07369d13ddd9fe508130c74ba601814: Status 404 returned error can't find the container with id 058e7c9af90ce1310f6268a5f053e545a07369d13ddd9fe508130c74ba601814
Nov 02 13:56:20.327327 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.327265    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"]
Nov 02 13:56:20.331274 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.331251    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.339482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.339461    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:20.339665 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.339541    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:56:20.342514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.342494    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"]
Nov 02 13:56:20.408699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.408662    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.408699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.408696    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wc8q\" (UniqueName: \"kubernetes.io/projected/17372a80-dcbb-4e43-8453-785c54ddbe68-kube-api-access-6wc8q\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.408920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.408728    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.509756 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.509719    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.509756 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.509755    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wc8q\" (UniqueName: \"kubernetes.io/projected/17372a80-dcbb-4e43-8453-785c54ddbe68-kube-api-access-6wc8q\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.510026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.509780    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.510425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.510401    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.512790 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.512766    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.521967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.521939    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wc8q\" (UniqueName: \"kubernetes.io/projected/17372a80-dcbb-4e43-8453-785c54ddbe68-kube-api-access-6wc8q\") pod \"maas-api-revoke-sub-15a941dd-kz7zc\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.581589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.581467    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" event={"ID":"59e91d94-7c4f-4d9e-ac04-04745b267604","Type":"ContainerStarted","Data":"4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf"}
Nov 02 13:56:20.581589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.581525    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" event={"ID":"59e91d94-7c4f-4d9e-ac04-04745b267604","Type":"ContainerStarted","Data":"058e7c9af90ce1310f6268a5f053e545a07369d13ddd9fe508130c74ba601814"}
Nov 02 13:56:20.603988 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.603929    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podStartSLOduration=1.603912236 podStartE2EDuration="1.603912236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:20.600840997 +0000 UTC m=+757.178929884" watchObservedRunningTime="2026-11-02 13:56:20.603912236 +0000 UTC m=+757.182001102"
Nov 02 13:56:20.649435 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.649399    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:20.928840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:20.928810    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"]
Nov 02 13:56:20.930099 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:20.930068    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17372a80_dcbb_4e43_8453_785c54ddbe68.slice/crio-17da1ce885343b1a892fd366d4bd2e3818a445f8f9ac028ec4b8ab98c26b9c18 WatchSource:0}: Error finding container 17da1ce885343b1a892fd366d4bd2e3818a445f8f9ac028ec4b8ab98c26b9c18: Status 404 returned error can't find the container with id 17da1ce885343b1a892fd366d4bd2e3818a445f8f9ac028ec4b8ab98c26b9c18
Nov 02 13:56:21.432359 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:56:21.432317    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:56:21.484309 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:21.484191    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:21.589312 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:21.589270    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc" event={"ID":"17372a80-dcbb-4e43-8453-785c54ddbe68","Type":"ContainerStarted","Data":"cea8fee5ac7c07b3e137ad8cf721592559051350bf96eebcb1b7ec459a5546de"}
Nov 02 13:56:21.589754 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:21.589341    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc" event={"ID":"17372a80-dcbb-4e43-8453-785c54ddbe68","Type":"ContainerStarted","Data":"17da1ce885343b1a892fd366d4bd2e3818a445f8f9ac028ec4b8ab98c26b9c18"}
Nov 02 13:56:21.609538 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:21.609486    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc" podStartSLOduration=1.609470127 podStartE2EDuration="1.609470127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:21.607362686 +0000 UTC m=+758.185451554" watchObservedRunningTime="2026-11-02 13:56:21.609470127 +0000 UTC m=+758.187558995"
Nov 02 13:56:22.596121 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.596086    5572 generic.go:356] "Generic (PLEG): container finished" podID="17372a80-dcbb-4e43-8453-785c54ddbe68" containerID="cea8fee5ac7c07b3e137ad8cf721592559051350bf96eebcb1b7ec459a5546de" exitCode=0
Nov 02 13:56:22.596529 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.596180    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc" event={"ID":"17372a80-dcbb-4e43-8453-785c54ddbe68","Type":"ContainerDied","Data":"cea8fee5ac7c07b3e137ad8cf721592559051350bf96eebcb1b7ec459a5546de"}
Nov 02 13:56:22.634929 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.634901    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"]
Nov 02 13:56:22.639320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.639297    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.648898 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.648872    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"]
Nov 02 13:56:22.742096 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.742064    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzh9b\" (UniqueName: \"kubernetes.io/projected/70905007-4f98-4865-8820-e3893d09b482-kube-api-access-qzh9b\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.742096 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.742109    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70905007-4f98-4865-8820-e3893d09b482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.742389 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.742277    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70905007-4f98-4865-8820-e3893d09b482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.843546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.843503    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzh9b\" (UniqueName: \"kubernetes.io/projected/70905007-4f98-4865-8820-e3893d09b482-kube-api-access-qzh9b\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.843546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.843586    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70905007-4f98-4865-8820-e3893d09b482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.843832 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.843664    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70905007-4f98-4865-8820-e3893d09b482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.844307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.844285    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70905007-4f98-4865-8820-e3893d09b482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.846188 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.846143    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70905007-4f98-4865-8820-e3893d09b482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.856227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.856196    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzh9b\" (UniqueName: \"kubernetes.io/projected/70905007-4f98-4865-8820-e3893d09b482-kube-api-access-qzh9b\") pod \"maas-api-revoke-sub-320446be-vffb6\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:22.955978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:22.955942    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:23.234931 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.234908    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"]
Nov 02 13:56:23.237092 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:23.237063    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70905007_4f98_4865_8820_e3893d09b482.slice/crio-e322aba5a2ff69dda2a563fdaaa4704081890b9b80dc10cee1eb2a68283e08ef WatchSource:0}: Error finding container e322aba5a2ff69dda2a563fdaaa4704081890b9b80dc10cee1eb2a68283e08ef: Status 404 returned error can't find the container with id e322aba5a2ff69dda2a563fdaaa4704081890b9b80dc10cee1eb2a68283e08ef
Nov 02 13:56:23.604278 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.604246    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6" event={"ID":"70905007-4f98-4865-8820-e3893d09b482","Type":"ContainerStarted","Data":"644256b1f9c2ab16ce03ba83c3f3e1a8186b42cd276aa4b7cb06ce5d0262dce5"}
Nov 02 13:56:23.604691 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.604299    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6" event={"ID":"70905007-4f98-4865-8820-e3893d09b482","Type":"ContainerStarted","Data":"e322aba5a2ff69dda2a563fdaaa4704081890b9b80dc10cee1eb2a68283e08ef"}
Nov 02 13:56:23.634092 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.634020    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6" podStartSLOduration=1.634000589 podStartE2EDuration="1.634000589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:23.628130533 +0000 UTC m=+760.206219402" watchObservedRunningTime="2026-11-02 13:56:23.634000589 +0000 UTC m=+760.212089458"
Nov 02 13:56:23.741181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.741150    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:23.865138 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.865100    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-openshift-ca\") pod \"17372a80-dcbb-4e43-8453-785c54ddbe68\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") "
Nov 02 13:56:23.865279 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.865235    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-xks-ca\") pod \"17372a80-dcbb-4e43-8453-785c54ddbe68\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") "
Nov 02 13:56:23.865327 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.865315    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17372a80-dcbb-4e43-8453-785c54ddbe68-kube-api-access-6wc8q\" (UniqueName: \"kubernetes.io/projected/17372a80-dcbb-4e43-8453-785c54ddbe68-kube-api-access-6wc8q\") pod \"17372a80-dcbb-4e43-8453-785c54ddbe68\" (UID: \"17372a80-dcbb-4e43-8453-785c54ddbe68\") "
Nov 02 13:56:23.865491 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.865465    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-openshift-ca" pod "17372a80-dcbb-4e43-8453-785c54ddbe68" (UID: "17372a80-dcbb-4e43-8453-785c54ddbe68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:23.865724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.865702    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:23.867459 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.867433    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-xks-ca" pod "17372a80-dcbb-4e43-8453-785c54ddbe68" (UID: "17372a80-dcbb-4e43-8453-785c54ddbe68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:23.867549 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.867516    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17372a80-dcbb-4e43-8453-785c54ddbe68-kube-api-access-6wc8q" pod "17372a80-dcbb-4e43-8453-785c54ddbe68" (UID: "17372a80-dcbb-4e43-8453-785c54ddbe68"). InnerVolumeSpecName "kube-api-access-6wc8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:23.966908 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.966875    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17372a80-dcbb-4e43-8453-785c54ddbe68-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:23.966908 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:23.966904    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wc8q\" (UniqueName: \"kubernetes.io/projected/17372a80-dcbb-4e43-8453-785c54ddbe68-kube-api-access-6wc8q\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:24.227579 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.227535    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"]
Nov 02 13:56:24.228166 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.228146    5572 cpu_manager.go:414] "containerMap: removing container" podUID="17372a80-dcbb-4e43-8453-785c54ddbe68" containerName="revoke-keys"
Nov 02 13:56:24.228166 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.228166    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17372a80-dcbb-4e43-8453-785c54ddbe68" containerName="revoke-keys"
Nov 02 13:56:24.228334 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.228304    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="17372a80-dcbb-4e43-8453-785c54ddbe68" containerName="revoke-keys"
Nov 02 13:56:24.231510 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.231487    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.240970 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.240942    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"]
Nov 02 13:56:24.371191 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.371144    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhlj9\" (UniqueName: \"kubernetes.io/projected/5aed93a7-8743-4919-a615-7a08384159b4-kube-api-access-vhlj9\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.371351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.371204    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.371415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.371371    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.472489 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.472401    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhlj9\" (UniqueName: \"kubernetes.io/projected/5aed93a7-8743-4919-a615-7a08384159b4-kube-api-access-vhlj9\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.472489 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.472443    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.472723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.472507    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.473135 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.473114    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.474994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.474974    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.494201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.494171    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhlj9\" (UniqueName: \"kubernetes.io/projected/5aed93a7-8743-4919-a615-7a08384159b4-kube-api-access-vhlj9\") pod \"maas-api-revoke-sub-a387011a-fdskf\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.571420 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.571385    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:24.630940 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.630889    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"
Nov 02 13:56:24.631337 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.630912    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc" event={"ID":"17372a80-dcbb-4e43-8453-785c54ddbe68","Type":"ContainerDied","Data":"17da1ce885343b1a892fd366d4bd2e3818a445f8f9ac028ec4b8ab98c26b9c18"}
Nov 02 13:56:24.631337 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.631066    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17da1ce885343b1a892fd366d4bd2e3818a445f8f9ac028ec4b8ab98c26b9c18"
Nov 02 13:56:24.633087 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.633057    5572 generic.go:356] "Generic (PLEG): container finished" podID="70905007-4f98-4865-8820-e3893d09b482" containerID="644256b1f9c2ab16ce03ba83c3f3e1a8186b42cd276aa4b7cb06ce5d0262dce5" exitCode=0
Nov 02 13:56:24.633199 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.633125    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6" event={"ID":"70905007-4f98-4865-8820-e3893d09b482","Type":"ContainerDied","Data":"644256b1f9c2ab16ce03ba83c3f3e1a8186b42cd276aa4b7cb06ce5d0262dce5"}
Nov 02 13:56:24.822705 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.822669    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"]
Nov 02 13:56:24.827777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.827756    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:24.840488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.840463    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"]
Nov 02 13:56:24.901755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.901726    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"]
Nov 02 13:56:24.902621 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:24.902588    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aed93a7_8743_4919_a615_7a08384159b4.slice/crio-e709e38cd4da68d66e0dc14233d564051b9f5715c3369953c0c7dfffa12a0b16 WatchSource:0}: Error finding container e709e38cd4da68d66e0dc14233d564051b9f5715c3369953c0c7dfffa12a0b16: Status 404 returned error can't find the container with id e709e38cd4da68d66e0dc14233d564051b9f5715c3369953c0c7dfffa12a0b16
Nov 02 13:56:24.984452 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.984412    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:24.984635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.984479    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2px\" (UniqueName: \"kubernetes.io/projected/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-kube-api-access-5h2px\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:24.984769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:24.984743    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.085484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.085405    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.085674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.085487    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2px\" (UniqueName: \"kubernetes.io/projected/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-kube-api-access-5h2px\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.085674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.085666    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.086230 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.086203    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.087894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.087875    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.097931 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.097907    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2px\" (UniqueName: \"kubernetes.io/projected/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-kube-api-access-5h2px\") pod \"maas-api-revoke-sub-b78b8466-pdmnc\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.142695 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.142660    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:25.423902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.423872    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"]
Nov 02 13:56:25.425402 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:25.425378    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7317ddd1_3735_46ad_a0b9_e1e6fcccc4b0.slice/crio-e441475919b39c033f3dddda7a5b96ab2b3cd888209e34f3c8b09c4c55ba47b3 WatchSource:0}: Error finding container e441475919b39c033f3dddda7a5b96ab2b3cd888209e34f3c8b09c4c55ba47b3: Status 404 returned error can't find the container with id e441475919b39c033f3dddda7a5b96ab2b3cd888209e34f3c8b09c4c55ba47b3
Nov 02 13:56:25.640142 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.640111    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc" event={"ID":"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0","Type":"ContainerStarted","Data":"e441475919b39c033f3dddda7a5b96ab2b3cd888209e34f3c8b09c4c55ba47b3"}
Nov 02 13:56:25.641859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.641831    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf" event={"ID":"5aed93a7-8743-4919-a615-7a08384159b4","Type":"ContainerStarted","Data":"4f12981cad13470917919216da973d2777f38c960e47a8636342b4c7399d9459"}
Nov 02 13:56:25.641995 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.641890    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf" event={"ID":"5aed93a7-8743-4919-a615-7a08384159b4","Type":"ContainerStarted","Data":"e709e38cd4da68d66e0dc14233d564051b9f5715c3369953c0c7dfffa12a0b16"}
Nov 02 13:56:25.660221 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.660123    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf" podStartSLOduration=1.660102725 podStartE2EDuration="1.660102725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:25.659812148 +0000 UTC m=+762.237901019" watchObservedRunningTime="2026-11-02 13:56:25.660102725 +0000 UTC m=+762.238191593"
Nov 02 13:56:25.786872 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.786848    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:25.900594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.899394    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70905007-4f98-4865-8820-e3893d09b482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70905007-4f98-4865-8820-e3893d09b482-maas-api-xks-ca\") pod \"70905007-4f98-4865-8820-e3893d09b482\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") "
Nov 02 13:56:25.900594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.899518    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70905007-4f98-4865-8820-e3893d09b482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70905007-4f98-4865-8820-e3893d09b482-maas-api-openshift-ca\") pod \"70905007-4f98-4865-8820-e3893d09b482\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") "
Nov 02 13:56:25.900594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.899718    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70905007-4f98-4865-8820-e3893d09b482-kube-api-access-qzh9b\" (UniqueName: \"kubernetes.io/projected/70905007-4f98-4865-8820-e3893d09b482-kube-api-access-qzh9b\") pod \"70905007-4f98-4865-8820-e3893d09b482\" (UID: \"70905007-4f98-4865-8820-e3893d09b482\") "
Nov 02 13:56:25.913035 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.908346    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70905007-4f98-4865-8820-e3893d09b482-maas-api-openshift-ca" pod "70905007-4f98-4865-8820-e3893d09b482" (UID: "70905007-4f98-4865-8820-e3893d09b482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:25.916427 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.913218    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70905007-4f98-4865-8820-e3893d09b482-maas-api-xks-ca" pod "70905007-4f98-4865-8820-e3893d09b482" (UID: "70905007-4f98-4865-8820-e3893d09b482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:25.936417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:25.936343    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70905007-4f98-4865-8820-e3893d09b482-kube-api-access-qzh9b" pod "70905007-4f98-4865-8820-e3893d09b482" (UID: "70905007-4f98-4865-8820-e3893d09b482"). InnerVolumeSpecName "kube-api-access-qzh9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:26.001591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.001539    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70905007-4f98-4865-8820-e3893d09b482-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:26.001591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.001591    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70905007-4f98-4865-8820-e3893d09b482-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:26.001837 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.001605    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzh9b\" (UniqueName: \"kubernetes.io/projected/70905007-4f98-4865-8820-e3893d09b482-kube-api-access-qzh9b\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:26.649425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.649389    5572 generic.go:356] "Generic (PLEG): container finished" podID="7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" containerID="dcf2aeb4bcf016d6b25e9ce269436cc1b6478af6f5d72003892ca4b68b3751b3" exitCode=0
Nov 02 13:56:26.649867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.649467    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc" event={"ID":"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0","Type":"ContainerDied","Data":"dcf2aeb4bcf016d6b25e9ce269436cc1b6478af6f5d72003892ca4b68b3751b3"}
Nov 02 13:56:26.650975 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.650945    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"
Nov 02 13:56:26.651118 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.650987    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6" event={"ID":"70905007-4f98-4865-8820-e3893d09b482","Type":"ContainerDied","Data":"e322aba5a2ff69dda2a563fdaaa4704081890b9b80dc10cee1eb2a68283e08ef"}
Nov 02 13:56:26.651118 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.651028    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e322aba5a2ff69dda2a563fdaaa4704081890b9b80dc10cee1eb2a68283e08ef"
Nov 02 13:56:26.652604 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.652583    5572 generic.go:356] "Generic (PLEG): container finished" podID="5aed93a7-8743-4919-a615-7a08384159b4" containerID="4f12981cad13470917919216da973d2777f38c960e47a8636342b4c7399d9459" exitCode=0
Nov 02 13:56:26.652748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:26.652609    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf" event={"ID":"5aed93a7-8743-4919-a615-7a08384159b4","Type":"ContainerDied","Data":"4f12981cad13470917919216da973d2777f38c960e47a8636342b4c7399d9459"}
Nov 02 13:56:27.821959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.821929    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:27.857430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.857404    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:27.927370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.927281    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-xks-ca\") pod \"5aed93a7-8743-4919-a615-7a08384159b4\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") "
Nov 02 13:56:27.927533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.927464    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-openshift-ca\") pod \"5aed93a7-8743-4919-a615-7a08384159b4\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") "
Nov 02 13:56:27.927533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.927497    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aed93a7-8743-4919-a615-7a08384159b4-kube-api-access-vhlj9\" (UniqueName: \"kubernetes.io/projected/5aed93a7-8743-4919-a615-7a08384159b4-kube-api-access-vhlj9\") pod \"5aed93a7-8743-4919-a615-7a08384159b4\" (UID: \"5aed93a7-8743-4919-a615-7a08384159b4\") "
Nov 02 13:56:27.927864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.927840    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-openshift-ca" pod "5aed93a7-8743-4919-a615-7a08384159b4" (UID: "5aed93a7-8743-4919-a615-7a08384159b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:27.929600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.929537    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-xks-ca" pod "5aed93a7-8743-4919-a615-7a08384159b4" (UID: "5aed93a7-8743-4919-a615-7a08384159b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:27.929720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:27.929642    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aed93a7-8743-4919-a615-7a08384159b4-kube-api-access-vhlj9" pod "5aed93a7-8743-4919-a615-7a08384159b4" (UID: "5aed93a7-8743-4919-a615-7a08384159b4"). InnerVolumeSpecName "kube-api-access-vhlj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:28.041590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.040490    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-xks-ca\") pod \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") "
Nov 02 13:56:28.041590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.040650    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-kube-api-access-5h2px\" (UniqueName: \"kubernetes.io/projected/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-kube-api-access-5h2px\") pod \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") "
Nov 02 13:56:28.041590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.040764    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-openshift-ca\") pod \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\" (UID: \"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0\") "
Nov 02 13:56:28.057633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.056310    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-openshift-ca" pod "7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" (UID: "7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:28.057633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.056507    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:28.057633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.056528    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:28.057633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.056542    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhlj9\" (UniqueName: \"kubernetes.io/projected/5aed93a7-8743-4919-a615-7a08384159b4-kube-api-access-vhlj9\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:28.057633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.056602    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aed93a7-8743-4919-a615-7a08384159b4-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:28.060783 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.060747    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-xks-ca" pod "7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" (UID: "7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:28.061283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.061254    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-kube-api-access-5h2px" pod "7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" (UID: "7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0"). InnerVolumeSpecName "kube-api-access-5h2px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:28.157780 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.157746    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:28.157780 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.157777    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h2px\" (UniqueName: \"kubernetes.io/projected/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0-kube-api-access-5h2px\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:28.666475 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.666443    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"
Nov 02 13:56:28.666714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.666473    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc" event={"ID":"7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0","Type":"ContainerDied","Data":"e441475919b39c033f3dddda7a5b96ab2b3cd888209e34f3c8b09c4c55ba47b3"}
Nov 02 13:56:28.666714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.666522    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e441475919b39c033f3dddda7a5b96ab2b3cd888209e34f3c8b09c4c55ba47b3"
Nov 02 13:56:28.667997 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.667980    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"
Nov 02 13:56:28.668086 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.668023    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf" event={"ID":"5aed93a7-8743-4919-a615-7a08384159b4","Type":"ContainerDied","Data":"e709e38cd4da68d66e0dc14233d564051b9f5715c3369953c0c7dfffa12a0b16"}
Nov 02 13:56:28.668086 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:28.668062    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e709e38cd4da68d66e0dc14233d564051b9f5715c3369953c0c7dfffa12a0b16"
Nov 02 13:56:29.484528 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:29.484496    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:29.497221 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:29.497198    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:29.686300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:29.686273    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:56:31.577811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.577766    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"]
Nov 02 13:56:31.578619 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578596    5572 cpu_manager.go:414] "containerMap: removing container" podUID="5aed93a7-8743-4919-a615-7a08384159b4" containerName="revoke-keys"
Nov 02 13:56:31.578719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578621    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aed93a7-8743-4919-a615-7a08384159b4" containerName="revoke-keys"
Nov 02 13:56:31.578719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578655    5572 cpu_manager.go:414] "containerMap: removing container" podUID="7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" containerName="revoke-keys"
Nov 02 13:56:31.578719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578664    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" containerName="revoke-keys"
Nov 02 13:56:31.578719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578713    5572 cpu_manager.go:414] "containerMap: removing container" podUID="70905007-4f98-4865-8820-e3893d09b482" containerName="revoke-keys"
Nov 02 13:56:31.578926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578721    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70905007-4f98-4865-8820-e3893d09b482" containerName="revoke-keys"
Nov 02 13:56:31.578926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578849    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="70905007-4f98-4865-8820-e3893d09b482" containerName="revoke-keys"
Nov 02 13:56:31.578926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578864    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aed93a7-8743-4919-a615-7a08384159b4" containerName="revoke-keys"
Nov 02 13:56:31.578926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.578876    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" containerName="revoke-keys"
Nov 02 13:56:31.583123 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.583100    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.591156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.590987    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:56:31.591156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.591116    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:31.596629 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.596599    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.615343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.607970    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"]
Nov 02 13:56:31.616008 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.615214    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.616167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.616091    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8q8k\" (UniqueName: \"kubernetes.io/projected/3c655716-836b-4a8f-a0bb-850101dc5526-kube-api-access-m8q8k\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.718165 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.718093    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.718346 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.718211    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.718346 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.718247    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q8k\" (UniqueName: \"kubernetes.io/projected/3c655716-836b-4a8f-a0bb-850101dc5526-kube-api-access-m8q8k\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.719192 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.719164    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.725357 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.725262    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.733098 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.733068    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q8k\" (UniqueName: \"kubernetes.io/projected/3c655716-836b-4a8f-a0bb-850101dc5526-kube-api-access-m8q8k\") pod \"maas-api-revoke-sub-e79d4fcc-lwhsv\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:31.921329 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:31.920712    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:32.564656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:32.556707    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"]
Nov 02 13:56:32.845734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:32.845640    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv" event={"ID":"3c655716-836b-4a8f-a0bb-850101dc5526","Type":"ContainerStarted","Data":"53408f5795c25550609ce4a86ccd911b64a6eb7cb2995141a212883d48684891"}
Nov 02 13:56:33.181724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.181631    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"]
Nov 02 13:56:33.219976 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.219931    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.274602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.252027    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f2e945\"/\"distinct-w1-f2e945-kserve-self-signed-certs\""
Nov 02 13:56:33.461648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.438274    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-model-cache\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.461971 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.461905    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-home\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.462159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.462139    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-dshm\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.462619 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.462552    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tls-certs\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.485593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.462795    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tmp-dir\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.488165 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.488137    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmh95\" (UniqueName: \"kubernetes.io/projected/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-kube-api-access-bmh95\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.521594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.486130    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:33.522824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.487243    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:33.531586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.486081    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:33.589304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.589269    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"]
Nov 02 13:56:33.594916 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.594552    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tmp-dir\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.595059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.594952    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tmp-dir\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.596134 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.595348    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmh95\" (UniqueName: \"kubernetes.io/projected/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-kube-api-access-bmh95\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.596134 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.595488    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-model-cache\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.596134 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.595520    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-home\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.596134 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.595578    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-dshm\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.596134 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.595647    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tls-certs\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.596764 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.596741    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-model-cache\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.597447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.597425    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-home\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.675588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.672118    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-dshm\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.683115 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.678588    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tls-certs\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.683115 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.680490    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:33.724642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.723184    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:33.734593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.724279    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:33.825285 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.825248    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmh95\" (UniqueName: \"kubernetes.io/projected/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-kube-api-access-bmh95\") pod \"distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") " pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:33.969591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:33.961261    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv" event={"ID":"3c655716-836b-4a8f-a0bb-850101dc5526","Type":"ContainerStarted","Data":"8ac3796dcdc6f4e2c53560612e5d7f02cff3ece6a39fd506aa98807545088c7e"}
Nov 02 13:56:34.016827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:34.016719    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:34.090596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:34.085546    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv" podStartSLOduration=3.085432829 podStartE2EDuration="3.085432829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:34.015300549 +0000 UTC m=+770.593389418" watchObservedRunningTime="2026-11-02 13:56:34.085432829 +0000 UTC m=+770.663521698"
Nov 02 13:56:34.675327 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:34.675241    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"]
Nov 02 13:56:34.686087 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:34.686055    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3ff03bf_bb7f_4611_8a37_b9533e028dd8.slice/crio-38818e402b556f70b30e9bd4b49d18928479fc4363885d2dc587942a8834f40b WatchSource:0}: Error finding container 38818e402b556f70b30e9bd4b49d18928479fc4363885d2dc587942a8834f40b: Status 404 returned error can't find the container with id 38818e402b556f70b30e9bd4b49d18928479fc4363885d2dc587942a8834f40b
Nov 02 13:56:34.974095 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:34.974052    5572 generic.go:356] "Generic (PLEG): container finished" podID="3c655716-836b-4a8f-a0bb-850101dc5526" containerID="8ac3796dcdc6f4e2c53560612e5d7f02cff3ece6a39fd506aa98807545088c7e" exitCode=0
Nov 02 13:56:34.974518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:34.974252    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv" event={"ID":"3c655716-836b-4a8f-a0bb-850101dc5526","Type":"ContainerDied","Data":"8ac3796dcdc6f4e2c53560612e5d7f02cff3ece6a39fd506aa98807545088c7e"}
Nov 02 13:56:34.976684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:34.976648    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" event={"ID":"d3ff03bf-bb7f-4611-8a37-b9533e028dd8","Type":"ContainerStarted","Data":"38818e402b556f70b30e9bd4b49d18928479fc4363885d2dc587942a8834f40b"}
Nov 02 13:56:35.022772 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.021806    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:35.076189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.076142    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podStartSLOduration=2.076123273 podStartE2EDuration="2.076123273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:35.075516254 +0000 UTC m=+771.653605122" watchObservedRunningTime="2026-11-02 13:56:35.076123273 +0000 UTC m=+771.654212140"
Nov 02 13:56:35.265895 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.265783    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"]
Nov 02 13:56:35.328585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.326052    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.405659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.405625    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"]
Nov 02 13:56:35.469642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.468308    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2k5\" (UniqueName: \"kubernetes.io/projected/ba1d57ed-ae4c-40ca-82ab-504da24f1308-kube-api-access-7j2k5\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.469642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.468580    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.469642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.468703    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.578588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.577202    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.578588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.577284    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2k5\" (UniqueName: \"kubernetes.io/projected/ba1d57ed-ae4c-40ca-82ab-504da24f1308-kube-api-access-7j2k5\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.578588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.577443    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.578588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.578142    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.639585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.628532    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.639585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.633069    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2k5\" (UniqueName: \"kubernetes.io/projected/ba1d57ed-ae4c-40ca-82ab-504da24f1308-kube-api-access-7j2k5\") pod \"maas-api-revoke-sub-741e4fcd-ctkcn\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.678590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.676951    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"]
Nov 02 13:56:35.776666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.769942    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"]
Nov 02 13:56:35.776666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.770150    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.856640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.856386    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908299    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908370    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908472    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908501    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5grz\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-kube-api-access-q5grz\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908582    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908639    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908666    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908732    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.909016 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.908794    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d021925-966b-4131-98b2-6b91d5ce6bc3-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:35.995770 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:35.995730    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" event={"ID":"d3ff03bf-bb7f-4611-8a37-b9533e028dd8","Type":"ContainerStarted","Data":"0ed31b4b7e74cf4e696d942d7b15088cfbe4516deac37b907e84b07accb435b8"}
Nov 02 13:56:36.010504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010461    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010522    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5grz\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-kube-api-access-q5grz\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010604    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010762 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010666    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010762 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010695    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010762    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010794    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d021925-966b-4131-98b2-6b91d5ce6bc3-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010883 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010869    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.010923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.010914    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.012105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.011922    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.012510 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.012373    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.012510 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.012425    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d021925-966b-4131-98b2-6b91d5ce6bc3-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.013001 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.012959    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.013290 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.013270    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.014766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.014740    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.021679 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.021649    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.043603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.043515    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.047192 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.047137    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5grz\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-kube-api-access-q5grz\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.136096 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.136018    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:36.167220 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.167196    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:36.179379 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.179352    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"]
Nov 02 13:56:36.213222 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.213191    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c655716-836b-4a8f-a0bb-850101dc5526-kube-api-access-m8q8k\" (UniqueName: \"kubernetes.io/projected/3c655716-836b-4a8f-a0bb-850101dc5526-kube-api-access-m8q8k\") pod \"3c655716-836b-4a8f-a0bb-850101dc5526\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") "
Nov 02 13:56:36.213366 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.213346    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-openshift-ca\") pod \"3c655716-836b-4a8f-a0bb-850101dc5526\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") "
Nov 02 13:56:36.213499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.213466    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-xks-ca\") pod \"3c655716-836b-4a8f-a0bb-850101dc5526\" (UID: \"3c655716-836b-4a8f-a0bb-850101dc5526\") "
Nov 02 13:56:36.214096 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.214073    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-openshift-ca" pod "3c655716-836b-4a8f-a0bb-850101dc5526" (UID: "3c655716-836b-4a8f-a0bb-850101dc5526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:36.216499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.216473    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-xks-ca" pod "3c655716-836b-4a8f-a0bb-850101dc5526" (UID: "3c655716-836b-4a8f-a0bb-850101dc5526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:36.216689 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.216668    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c655716-836b-4a8f-a0bb-850101dc5526-kube-api-access-m8q8k" pod "3c655716-836b-4a8f-a0bb-850101dc5526" (UID: "3c655716-836b-4a8f-a0bb-850101dc5526"). InnerVolumeSpecName "kube-api-access-m8q8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:36.314936 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.314903    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:36.314936 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.314933    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8q8k\" (UniqueName: \"kubernetes.io/projected/3c655716-836b-4a8f-a0bb-850101dc5526-kube-api-access-m8q8k\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:36.314936 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.314942    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c655716-836b-4a8f-a0bb-850101dc5526-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:36.424547 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.424522    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"]
Nov 02 13:56:36.483366 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:36.483325    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d021925_966b_4131_98b2_6b91d5ce6bc3.slice/crio-2d95c70493a9388a3ca8da5f460bbb5c762a85a4eb07624b8a546a778fb38c7a WatchSource:0}: Error finding container 2d95c70493a9388a3ca8da5f460bbb5c762a85a4eb07624b8a546a778fb38c7a: Status 404 returned error can't find the container with id 2d95c70493a9388a3ca8da5f460bbb5c762a85a4eb07624b8a546a778fb38c7a
Nov 02 13:56:36.486239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.486207    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:36.486314 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.486300    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:36.486347 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:36.486329    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:37.002920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.002883    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn" event={"ID":"ba1d57ed-ae4c-40ca-82ab-504da24f1308","Type":"ContainerStarted","Data":"ca50b15efbeaad7b708207ba47cb3e6fa69c32ef785f37e22812af638b4997f2"}
Nov 02 13:56:37.002920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.002939    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn" event={"ID":"ba1d57ed-ae4c-40ca-82ab-504da24f1308","Type":"ContainerStarted","Data":"831be56fe851ffed0e2629db6699f70ee1a640da171c0e6343fda87f1de86611"}
Nov 02 13:56:37.004616 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.004591    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" event={"ID":"4d021925-966b-4131-98b2-6b91d5ce6bc3","Type":"ContainerStarted","Data":"f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887"}
Nov 02 13:56:37.004724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.004632    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" event={"ID":"4d021925-966b-4131-98b2-6b91d5ce6bc3","Type":"ContainerStarted","Data":"2d95c70493a9388a3ca8da5f460bbb5c762a85a4eb07624b8a546a778fb38c7a"}
Nov 02 13:56:37.005954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.005937    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"
Nov 02 13:56:37.006024 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.005975    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv" event={"ID":"3c655716-836b-4a8f-a0bb-850101dc5526","Type":"ContainerDied","Data":"53408f5795c25550609ce4a86ccd911b64a6eb7cb2995141a212883d48684891"}
Nov 02 13:56:37.006024 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.006007    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53408f5795c25550609ce4a86ccd911b64a6eb7cb2995141a212883d48684891"
Nov 02 13:56:37.077527 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.077475    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn" podStartSLOduration=2.077461404 podStartE2EDuration="2.077461404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:37.075000383 +0000 UTC m=+773.653089250" watchObservedRunningTime="2026-11-02 13:56:37.077461404 +0000 UTC m=+773.655550270"
Nov 02 13:56:37.114846 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.114794    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podStartSLOduration=2.114777359 podStartE2EDuration="2.114777359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:37.11284688 +0000 UTC m=+773.690935748" watchObservedRunningTime="2026-11-02 13:56:37.114777359 +0000 UTC m=+773.692866225"
Nov 02 13:56:37.137121 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.137056    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:37.138955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.138926    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 13:56:37.139068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:37.138973    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 13:56:38.015867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:38.015832    5572 generic.go:356] "Generic (PLEG): container finished" podID="ba1d57ed-ae4c-40ca-82ab-504da24f1308" containerID="ca50b15efbeaad7b708207ba47cb3e6fa69c32ef785f37e22812af638b4997f2" exitCode=0
Nov 02 13:56:38.016334 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:38.015917    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn" event={"ID":"ba1d57ed-ae4c-40ca-82ab-504da24f1308","Type":"ContainerDied","Data":"ca50b15efbeaad7b708207ba47cb3e6fa69c32ef785f37e22812af638b4997f2"}
Nov 02 13:56:38.137426 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:38.137390    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 13:56:38.137629 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:38.137460    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 13:56:39.137246 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.137215    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 13:56:39.137659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.137274    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 13:56:39.229236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.229207    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:39.357793 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.357751    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1d57ed-ae4c-40ca-82ab-504da24f1308-kube-api-access-7j2k5\" (UniqueName: \"kubernetes.io/projected/ba1d57ed-ae4c-40ca-82ab-504da24f1308-kube-api-access-7j2k5\") pod \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") "
Nov 02 13:56:39.357983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.357865    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-openshift-ca\") pod \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") "
Nov 02 13:56:39.358058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.357996    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-xks-ca\") pod \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\" (UID: \"ba1d57ed-ae4c-40ca-82ab-504da24f1308\") "
Nov 02 13:56:39.358495 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.358266    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-openshift-ca" pod "ba1d57ed-ae4c-40ca-82ab-504da24f1308" (UID: "ba1d57ed-ae4c-40ca-82ab-504da24f1308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:39.358495 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.358470    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:39.360240 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.360209    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-xks-ca" pod "ba1d57ed-ae4c-40ca-82ab-504da24f1308" (UID: "ba1d57ed-ae4c-40ca-82ab-504da24f1308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:39.360637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.360610    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1d57ed-ae4c-40ca-82ab-504da24f1308-kube-api-access-7j2k5" pod "ba1d57ed-ae4c-40ca-82ab-504da24f1308" (UID: "ba1d57ed-ae4c-40ca-82ab-504da24f1308"). InnerVolumeSpecName "kube-api-access-7j2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:39.461592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.460378    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j2k5\" (UniqueName: \"kubernetes.io/projected/ba1d57ed-ae4c-40ca-82ab-504da24f1308-kube-api-access-7j2k5\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:39.461592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:39.460417    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1d57ed-ae4c-40ca-82ab-504da24f1308-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:40.031147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:40.031117    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"
Nov 02 13:56:40.031147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:40.031143    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn" event={"ID":"ba1d57ed-ae4c-40ca-82ab-504da24f1308","Type":"ContainerDied","Data":"831be56fe851ffed0e2629db6699f70ee1a640da171c0e6343fda87f1de86611"}
Nov 02 13:56:40.031538 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:40.031185    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831be56fe851ffed0e2629db6699f70ee1a640da171c0e6343fda87f1de86611"
Nov 02 13:56:40.136584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:40.136526    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 13:56:40.136810 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:40.136615    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 13:56:41.137257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.137218    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 13:56:41.137713 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.137430    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 13:56:41.961913 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.961876    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"]
Nov 02 13:56:41.962421 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.962409    5572 cpu_manager.go:414] "containerMap: removing container" podUID="ba1d57ed-ae4c-40ca-82ab-504da24f1308" containerName="revoke-keys"
Nov 02 13:56:41.962465 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.962423    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1d57ed-ae4c-40ca-82ab-504da24f1308" containerName="revoke-keys"
Nov 02 13:56:41.962465 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.962442    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3c655716-836b-4a8f-a0bb-850101dc5526" containerName="revoke-keys"
Nov 02 13:56:41.962465 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.962449    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c655716-836b-4a8f-a0bb-850101dc5526" containerName="revoke-keys"
Nov 02 13:56:41.962592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.962516    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1d57ed-ae4c-40ca-82ab-504da24f1308" containerName="revoke-keys"
Nov 02 13:56:41.962592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.962527    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c655716-836b-4a8f-a0bb-850101dc5526" containerName="revoke-keys"
Nov 02 13:56:41.966396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.966380    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:41.972770 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.972744    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:41.972932 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.972914    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:56:42.009591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:41.998343    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"]
Nov 02 13:56:42.117984 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.117948    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.118166 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.118092    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.118166 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.118120    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdsdn\" (UniqueName: \"kubernetes.io/projected/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-kube-api-access-gdsdn\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.137050 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.137016    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 13:56:42.137232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.137077    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 13:56:42.219828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.219723    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.220289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.219855    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.220289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.219880    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdsdn\" (UniqueName: \"kubernetes.io/projected/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-kube-api-access-gdsdn\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.220499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.220477    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.222262 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.222233    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.232684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.232664    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdsdn\" (UniqueName: \"kubernetes.io/projected/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-kube-api-access-gdsdn\") pod \"maas-api-revoke-sub-e92d454f-6zffn\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.313903 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.313867    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:42.610959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:42.610922    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"]
Nov 02 13:56:42.612092 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:42.611999    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cd6d80_eee2_486c_8b88_8f6c6b84f7d7.slice/crio-cf4534b12ab19414786bec69406e83d1f5961f4de02c34417353b958b4a581ff WatchSource:0}: Error finding container cf4534b12ab19414786bec69406e83d1f5961f4de02c34417353b958b4a581ff: Status 404 returned error can't find the container with id cf4534b12ab19414786bec69406e83d1f5961f4de02c34417353b958b4a581ff
Nov 02 13:56:43.052548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:43.052512    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn" event={"ID":"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7","Type":"ContainerStarted","Data":"03673cb442edeb25efba260484a2ed7b646dae059fa5f4d6a8d9b8a227c9cdcf"}
Nov 02 13:56:43.052548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:43.052587    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn" event={"ID":"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7","Type":"ContainerStarted","Data":"cf4534b12ab19414786bec69406e83d1f5961f4de02c34417353b958b4a581ff"}
Nov 02 13:56:43.073345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:43.073290    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn" podStartSLOduration=2.073274621 podStartE2EDuration="2.073274621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:43.070208079 +0000 UTC m=+779.648296948" watchObservedRunningTime="2026-11-02 13:56:43.073274621 +0000 UTC m=+779.651363489"
Nov 02 13:56:43.137251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:43.137210    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 13:56:43.137398 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:43.137278    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 13:56:44.017471 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.017436    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:44.030372 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.030346    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:44.060372 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.060343    5572 generic.go:356] "Generic (PLEG): container finished" podID="98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" containerID="03673cb442edeb25efba260484a2ed7b646dae059fa5f4d6a8d9b8a227c9cdcf" exitCode=0
Nov 02 13:56:44.060590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.060430    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn" event={"ID":"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7","Type":"ContainerDied","Data":"03673cb442edeb25efba260484a2ed7b646dae059fa5f4d6a8d9b8a227c9cdcf"}
Nov 02 13:56:44.072090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.072060    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:56:44.141238 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.141207    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:44.141637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.141615    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:44.142334 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:44.142307    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 13:56:45.236017 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.235985    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:45.377100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.377019    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-xks-ca\") pod \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") "
Nov 02 13:56:45.377100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.377064    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-openshift-ca\") pod \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") "
Nov 02 13:56:45.377100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.377094    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-kube-api-access-gdsdn\" (UniqueName: \"kubernetes.io/projected/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-kube-api-access-gdsdn\") pod \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\" (UID: \"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7\") "
Nov 02 13:56:45.377508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.377477    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-openshift-ca" pod "98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" (UID: "98cd6d80-eee2-486c-8b88-8f6c6b84f7d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:45.379299 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.379272    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-xks-ca" pod "98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" (UID: "98cd6d80-eee2-486c-8b88-8f6c6b84f7d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:45.379408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.379372    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-kube-api-access-gdsdn" pod "98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" (UID: "98cd6d80-eee2-486c-8b88-8f6c6b84f7d7"). InnerVolumeSpecName "kube-api-access-gdsdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:45.478880 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.478853    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:45.478880 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.478878    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:45.479033 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:45.478888    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdsdn\" (UniqueName: \"kubernetes.io/projected/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7-kube-api-access-gdsdn\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:46.074725 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:46.074688    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn" event={"ID":"98cd6d80-eee2-486c-8b88-8f6c6b84f7d7","Type":"ContainerDied","Data":"cf4534b12ab19414786bec69406e83d1f5961f4de02c34417353b958b4a581ff"}
Nov 02 13:56:46.074903 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:46.074739    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf4534b12ab19414786bec69406e83d1f5961f4de02c34417353b958b4a581ff"
Nov 02 13:56:46.074903 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:46.074705    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"
Nov 02 13:56:47.385818 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.385787    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"]
Nov 02 13:56:47.386636 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.386616    5572 cpu_manager.go:414] "containerMap: removing container" podUID="98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" containerName="revoke-keys"
Nov 02 13:56:47.386738 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.386637    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" containerName="revoke-keys"
Nov 02 13:56:47.386811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.386799    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" containerName="revoke-keys"
Nov 02 13:56:47.391727 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.391707    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.400766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.400173    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-model-cache\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.400766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.400237    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ckx\" (UniqueName: \"kubernetes.io/projected/3120bf68-fa50-4b09-9725-be8ed6134529-kube-api-access-w7ckx\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.400766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.400408    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3120bf68-fa50-4b09-9725-be8ed6134529-tls-certs\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.400766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.400512    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-tmp-dir\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.400766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.400636    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-dshm\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.400766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.400713    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-home\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.401840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.401807    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ee7df2\"/\"distinct-2-w2-ee7df2-kserve-self-signed-certs\""
Nov 02 13:56:47.404214 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.404184    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"]
Nov 02 13:56:47.501214 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501189    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3120bf68-fa50-4b09-9725-be8ed6134529-tls-certs\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501381 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501282    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-tmp-dir\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501381 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501329    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-dshm\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501381    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-home\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501442    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-model-cache\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501490    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ckx\" (UniqueName: \"kubernetes.io/projected/3120bf68-fa50-4b09-9725-be8ed6134529-kube-api-access-w7ckx\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501745    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-tmp-dir\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501829 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501806    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-home\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.501895 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.501843    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-model-cache\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.503706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.503688    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-dshm\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.503834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.503819    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3120bf68-fa50-4b09-9725-be8ed6134529-tls-certs\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.579368 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.579294    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ckx\" (UniqueName: \"kubernetes.io/projected/3120bf68-fa50-4b09-9725-be8ed6134529-kube-api-access-w7ckx\") pod \"distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") " pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:47.707684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:47.707611    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:48.000031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:48.000006    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"]
Nov 02 13:56:48.000614 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:48.000587    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3120bf68_fa50_4b09_9725_be8ed6134529.slice/crio-d6cf91b32fd79479c7f3b3220cf0b94eb3be64a581ea110eb2275f54124d899a WatchSource:0}: Error finding container d6cf91b32fd79479c7f3b3220cf0b94eb3be64a581ea110eb2275f54124d899a: Status 404 returned error can't find the container with id d6cf91b32fd79479c7f3b3220cf0b94eb3be64a581ea110eb2275f54124d899a
Nov 02 13:56:48.089051 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:48.089025    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" event={"ID":"3120bf68-fa50-4b09-9725-be8ed6134529","Type":"ContainerStarted","Data":"0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b"}
Nov 02 13:56:48.089267 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:48.089061    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" event={"ID":"3120bf68-fa50-4b09-9725-be8ed6134529","Type":"ContainerStarted","Data":"d6cf91b32fd79479c7f3b3220cf0b94eb3be64a581ea110eb2275f54124d899a"}
Nov 02 13:56:48.122828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:48.119487    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podStartSLOduration=1.119469433 podStartE2EDuration="1.119469433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:48.118937057 +0000 UTC m=+784.697025924" watchObservedRunningTime="2026-11-02 13:56:48.119469433 +0000 UTC m=+784.697558302"
Nov 02 13:56:48.718583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:48.718381    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:50.831103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.831070    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"]
Nov 02 13:56:50.835350 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.835332    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.849160 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.849134    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"]
Nov 02 13:56:50.894938 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.894904    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.894962    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.895053    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895177 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.895121    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895219 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.895177    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67594d7c-a1c4-4e31-bc0f-a635116017b1-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895219 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.895198    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkwk\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-kube-api-access-vwkwk\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895285 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.895220    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895318 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.895301    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.895352 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.895331    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.925959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.925920    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"]
Nov 02 13:56:50.930079 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.930057    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:50.934605 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.934473    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:50.934911 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.934860    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:56:50.950791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.939592    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"]
Nov 02 13:56:50.996877 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.996840    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.996905    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:50.997140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.996957    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.996987    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997067    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997143    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:50.997400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997199    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6c8\" (UniqueName: \"kubernetes.io/projected/d5b54d20-df49-4c6d-ab61-5849a8932ac5-kube-api-access-tc6c8\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:50.997504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997405    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67594d7c-a1c4-4e31-bc0f-a635116017b1-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997447    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkwk\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-kube-api-access-vwkwk\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997482    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997524    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997534    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997629    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997636    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997899 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997826    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.997974 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.997950    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.998215 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.998196    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67594d7c-a1c4-4e31-bc0f-a635116017b1-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:50.999926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:50.999895    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:51.000106 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.000085    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:51.015075 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.015041    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"]
Nov 02 13:56:51.018649 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.018624    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:51.020227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.020210    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.021457 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.021439    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkwk\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-kube-api-access-vwkwk\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:51.034017 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.033989    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"]
Nov 02 13:56:51.115839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.111321    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgmf\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-kube-api-access-2xgmf\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116198 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116151    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:51.116400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116217    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116278    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6c8\" (UniqueName: \"kubernetes.io/projected/d5b54d20-df49-4c6d-ab61-5849a8932ac5-kube-api-access-tc6c8\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:51.116400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116321    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116366    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465e3231-6a1e-46f4-bdac-2e25cf860191-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116747 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116468    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116747 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116517    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116747 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116621    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116747 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116687    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116964 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116780    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.116964 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.116825    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:51.118388 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.118352    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:51.163602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.151247    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:51.163854 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.151471    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:51.164281 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.154053    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6c8\" (UniqueName: \"kubernetes.io/projected/d5b54d20-df49-4c6d-ab61-5849a8932ac5-kube-api-access-tc6c8\") pod \"maas-api-revoke-sub-8c9a12b1-xm67m\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:51.218383 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218347    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218513 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218470    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgmf\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-kube-api-access-2xgmf\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218513 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218505    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218537    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218581    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465e3231-6a1e-46f4-bdac-2e25cf860191-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218644    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218808 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218679    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218808 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218743    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.218808 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.218781    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.221178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.220156    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465e3231-6a1e-46f4-bdac-2e25cf860191-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.221178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.220905    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.221178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.221057    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.221178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.221134    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.221421 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.221283    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.222616 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.222596    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.223231 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.223211    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.235859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.235799    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgmf\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-kube-api-access-2xgmf\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.238119 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.238079    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.273105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.273070    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:51.351946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.351907    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:51.441242 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:56:51.441129    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:56:51.456979 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.456939    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"]
Nov 02 13:56:51.458994 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:51.458954    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67594d7c_a1c4_4e31_bc0f_a635116017b1.slice/crio-227a5ae1b0f004f76ed03a368ff7fdaa872dfa6f9dbfa183e28beb2357734638 WatchSource:0}: Error finding container 227a5ae1b0f004f76ed03a368ff7fdaa872dfa6f9dbfa183e28beb2357734638: Status 404 returned error can't find the container with id 227a5ae1b0f004f76ed03a368ff7fdaa872dfa6f9dbfa183e28beb2357734638
Nov 02 13:56:51.461942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.461903    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:51.462023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.461993    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:51.462074 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.462041    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:51.560973 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.560937    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"]
Nov 02 13:56:51.562017 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:51.561986    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b54d20_df49_4c6d_ab61_5849a8932ac5.slice/crio-f86e726f8171fda7647397fbb52475612b4e109ffa5e0d2b8e899ad73f9b7bd5 WatchSource:0}: Error finding container f86e726f8171fda7647397fbb52475612b4e109ffa5e0d2b8e899ad73f9b7bd5: Status 404 returned error can't find the container with id f86e726f8171fda7647397fbb52475612b4e109ffa5e0d2b8e899ad73f9b7bd5
Nov 02 13:56:51.647698 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:51.647665    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465e3231_6a1e_46f4_bdac_2e25cf860191.slice/crio-503c8e1bffbd98c1e438e2d970144bdf30a692000d5b39b4e11de8ed1012016e WatchSource:0}: Error finding container 503c8e1bffbd98c1e438e2d970144bdf30a692000d5b39b4e11de8ed1012016e: Status 404 returned error can't find the container with id 503c8e1bffbd98c1e438e2d970144bdf30a692000d5b39b4e11de8ed1012016e
Nov 02 13:56:51.648061 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.648038    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"]
Nov 02 13:56:51.651037 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.651002    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:51.651153 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.651089    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:51.651153 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:51.651136    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:56:52.178882 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.178778    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" event={"ID":"465e3231-6a1e-46f4-bdac-2e25cf860191","Type":"ContainerStarted","Data":"bb718e678cab0068a58dc31e681339c857ff1ee566e2b93e7a259d53c958bd24"}
Nov 02 13:56:52.179310 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.178884    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" event={"ID":"465e3231-6a1e-46f4-bdac-2e25cf860191","Type":"ContainerStarted","Data":"503c8e1bffbd98c1e438e2d970144bdf30a692000d5b39b4e11de8ed1012016e"}
Nov 02 13:56:52.213045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.212997    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m" event={"ID":"d5b54d20-df49-4c6d-ab61-5849a8932ac5","Type":"ContainerStarted","Data":"00b78bf55b98ed205b68d6cc4ef41188d69ec55b73401aa902ae5d883fe29382"}
Nov 02 13:56:52.213349 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.213316    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m" event={"ID":"d5b54d20-df49-4c6d-ab61-5849a8932ac5","Type":"ContainerStarted","Data":"f86e726f8171fda7647397fbb52475612b4e109ffa5e0d2b8e899ad73f9b7bd5"}
Nov 02 13:56:52.215160 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.215130    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" event={"ID":"67594d7c-a1c4-4e31-bc0f-a635116017b1","Type":"ContainerStarted","Data":"6b829521a0fa31b14a586d84abe04daabcef0ad5564b38d3fd9f8994f7dde93c"}
Nov 02 13:56:52.215263 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.215171    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" event={"ID":"67594d7c-a1c4-4e31-bc0f-a635116017b1","Type":"ContainerStarted","Data":"227a5ae1b0f004f76ed03a368ff7fdaa872dfa6f9dbfa183e28beb2357734638"}
Nov 02 13:56:52.219770 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.219725    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podStartSLOduration=2.21971168 podStartE2EDuration="2.21971168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:52.216661891 +0000 UTC m=+788.794750759" watchObservedRunningTime="2026-11-02 13:56:52.21971168 +0000 UTC m=+788.797800546"
Nov 02 13:56:52.243226 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.243166    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podStartSLOduration=2.243149642 podStartE2EDuration="2.243149642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:52.2406302 +0000 UTC m=+788.818719068" watchObservedRunningTime="2026-11-02 13:56:52.243149642 +0000 UTC m=+788.821238510"
Nov 02 13:56:52.267382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.267323    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m" podStartSLOduration=2.267305633 podStartE2EDuration="2.267305633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:52.265118336 +0000 UTC m=+788.843207204" watchObservedRunningTime="2026-11-02 13:56:52.267305633 +0000 UTC m=+788.845394500"
Nov 02 13:56:52.354752 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.354715    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 13:56:52.355030 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.355000    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 13:56:52.355401 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:52.355380    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:53.152166 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:53.152128    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:53.153631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:53.153605    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 13:56:53.153779 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:53.153660    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 13:56:53.240310 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:53.240276    5572 generic.go:356] "Generic (PLEG): container finished" podID="d5b54d20-df49-4c6d-ab61-5849a8932ac5" containerID="00b78bf55b98ed205b68d6cc4ef41188d69ec55b73401aa902ae5d883fe29382" exitCode=0
Nov 02 13:56:53.240708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:53.240336    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m" event={"ID":"d5b54d20-df49-4c6d-ab61-5849a8932ac5","Type":"ContainerDied","Data":"00b78bf55b98ed205b68d6cc4ef41188d69ec55b73401aa902ae5d883fe29382"}
Nov 02 13:56:53.364889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:53.362899    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 13:56:53.364889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:53.362957    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 13:56:54.152778 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.152747    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 13:56:54.152953 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.152798    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 13:56:54.352489 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.352261    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"]
Nov 02 13:56:54.354963 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.352677    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 13:56:54.355086 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.354992    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 13:56:54.357226 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.357205    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.369787 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.369769    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1d18fb\"/\"distinct-w5-1d18fb-kserve-self-signed-certs\""
Nov 02 13:56:54.373047 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.372662    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"]
Nov 02 13:56:54.374250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.373423    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-model-cache\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.374250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.373599    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-tmp-dir\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.374250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.373722    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-home\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.374250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.373753    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c246434-da6e-4c64-8743-b3778b117e7a-tls-certs\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.374250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.373776    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gwp\" (UniqueName: \"kubernetes.io/projected/5c246434-da6e-4c64-8743-b3778b117e7a-kube-api-access-65gwp\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.374250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.373827    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-dshm\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.475648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.475549    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-home\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.475648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.475616    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c246434-da6e-4c64-8743-b3778b117e7a-tls-certs\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.475648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.475644    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gwp\" (UniqueName: \"kubernetes.io/projected/5c246434-da6e-4c64-8743-b3778b117e7a-kube-api-access-65gwp\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.475918 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.475696    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-dshm\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.475918 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.475775    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-model-cache\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.475918 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.475912    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-tmp-dir\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.476075 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.475974    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-home\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.476391 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.476367    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-tmp-dir\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.476639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.476620    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-model-cache\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.479610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.479279    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-dshm\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.479610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.479325    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c246434-da6e-4c64-8743-b3778b117e7a-tls-certs\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.548598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.546637    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gwp\" (UniqueName: \"kubernetes.io/projected/5c246434-da6e-4c64-8743-b3778b117e7a-kube-api-access-65gwp\") pod \"distinct-w5-1d18fb-kserve-6857c74968-989cs\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") " pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.652723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.652365    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:54.690583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.687144    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-openshift-ca\") pod \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") "
Nov 02 13:56:54.690583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.687329    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b54d20-df49-4c6d-ab61-5849a8932ac5-kube-api-access-tc6c8\" (UniqueName: \"kubernetes.io/projected/d5b54d20-df49-4c6d-ab61-5849a8932ac5-kube-api-access-tc6c8\") pod \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") "
Nov 02 13:56:54.690583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.687474    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-xks-ca\") pod \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\" (UID: \"d5b54d20-df49-4c6d-ab61-5849a8932ac5\") "
Nov 02 13:56:54.705638 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.693950    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-openshift-ca" pod "d5b54d20-df49-4c6d-ab61-5849a8932ac5" (UID: "d5b54d20-df49-4c6d-ab61-5849a8932ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:54.748486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.747157    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:54.847701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.846806    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:54.848508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.848461    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b54d20-df49-4c6d-ab61-5849a8932ac5-kube-api-access-tc6c8" pod "d5b54d20-df49-4c6d-ab61-5849a8932ac5" (UID: "d5b54d20-df49-4c6d-ab61-5849a8932ac5"). InnerVolumeSpecName "kube-api-access-tc6c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:54.848708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.848687    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-xks-ca" pod "d5b54d20-df49-4c6d-ab61-5849a8932ac5" (UID: "d5b54d20-df49-4c6d-ab61-5849a8932ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:54.951147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.950082    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b54d20-df49-4c6d-ab61-5849a8932ac5-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:54.951147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:54.950106    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc6c8\" (UniqueName: \"kubernetes.io/projected/d5b54d20-df49-4c6d-ab61-5849a8932ac5-kube-api-access-tc6c8\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:56:55.152893 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.152860    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 13:56:55.153041 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.152919    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 13:56:55.176014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.175990    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"]
Nov 02 13:56:55.177736 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:55.177712    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c246434_da6e_4c64_8743_b3778b117e7a.slice/crio-628de5c73493baff479eef4cd589cbbcaee9a41337efab6e03853abcc46227a7 WatchSource:0}: Error finding container 628de5c73493baff479eef4cd589cbbcaee9a41337efab6e03853abcc46227a7: Status 404 returned error can't find the container with id 628de5c73493baff479eef4cd589cbbcaee9a41337efab6e03853abcc46227a7
Nov 02 13:56:55.263254 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.263228    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"
Nov 02 13:56:55.263792 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.263758    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m" event={"ID":"d5b54d20-df49-4c6d-ab61-5849a8932ac5","Type":"ContainerDied","Data":"f86e726f8171fda7647397fbb52475612b4e109ffa5e0d2b8e899ad73f9b7bd5"}
Nov 02 13:56:55.263915 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.263849    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f86e726f8171fda7647397fbb52475612b4e109ffa5e0d2b8e899ad73f9b7bd5"
Nov 02 13:56:55.264801 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.264777    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" event={"ID":"5c246434-da6e-4c64-8743-b3778b117e7a","Type":"ContainerStarted","Data":"628de5c73493baff479eef4cd589cbbcaee9a41337efab6e03853abcc46227a7"}
Nov 02 13:56:55.353236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.353124    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 13:56:55.353236 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:55.353182    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 13:56:56.152521 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.152480    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 13:56:56.152733 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.152542    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 13:56:56.273247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.273210    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" event={"ID":"5c246434-da6e-4c64-8743-b3778b117e7a","Type":"ContainerStarted","Data":"26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3"}
Nov 02 13:56:56.311708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.311647    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podStartSLOduration=2.311628065 podStartE2EDuration="2.311628065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:56.310911001 +0000 UTC m=+792.888999869" watchObservedRunningTime="2026-11-02 13:56:56.311628065 +0000 UTC m=+792.889716933"
Nov 02 13:56:56.353089 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.353057    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 13:56:56.353251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.353118    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 13:56:56.567597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.565986    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"]
Nov 02 13:56:56.567597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.566940    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d5b54d20-df49-4c6d-ab61-5849a8932ac5" containerName="revoke-keys"
Nov 02 13:56:56.567597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.566962    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b54d20-df49-4c6d-ab61-5849a8932ac5" containerName="revoke-keys"
Nov 02 13:56:56.567597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.567132    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b54d20-df49-4c6d-ab61-5849a8932ac5" containerName="revoke-keys"
Nov 02 13:56:56.667583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.666857    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"]
Nov 02 13:56:56.667583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.666988    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.672501 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.672118    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:56:56.672501 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.672213    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:56.758451 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.758423    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:56:56.807598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.805242    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9mb\" (UniqueName: \"kubernetes.io/projected/2a0f1ebd-01d2-4ed1-914c-1657a788291d-kube-api-access-mc9mb\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.807598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.805343    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.807598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.805433    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.909641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.909324    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9mb\" (UniqueName: \"kubernetes.io/projected/2a0f1ebd-01d2-4ed1-914c-1657a788291d-kube-api-access-mc9mb\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.909641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.909430    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.909641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.909519    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.913588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.910299    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.936251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.936211    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:56.937990 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:56.937964    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9mb\" (UniqueName: \"kubernetes.io/projected/2a0f1ebd-01d2-4ed1-914c-1657a788291d-kube-api-access-mc9mb\") pod \"maas-api-revoke-sub-f5455a7b-mrmgj\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:57.039788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.039755    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:56:57.153010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.152100    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 13:56:57.153010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.152195    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 13:56:57.357376 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.356297    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 13:56:57.357376 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.356354    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 13:56:57.418927 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.417638    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"]
Nov 02 13:56:57.441711 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:56:57.441683    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0f1ebd_01d2_4ed1_914c_1657a788291d.slice/crio-8745e7199b4497002aa950588361ba5a2ef1b3e3fc2564a4294eb1b2c266a2cd WatchSource:0}: Error finding container 8745e7199b4497002aa950588361ba5a2ef1b3e3fc2564a4294eb1b2c266a2cd: Status 404 returned error can't find the container with id 8745e7199b4497002aa950588361ba5a2ef1b3e3fc2564a4294eb1b2c266a2cd
Nov 02 13:56:57.708549 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.708526    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:57.727817 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:57.727796    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:58.152212 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.152169    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 13:56:58.152394 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.152242    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 13:56:58.311140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.311096    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj" event={"ID":"2a0f1ebd-01d2-4ed1-914c-1657a788291d","Type":"ContainerStarted","Data":"c37a91ef9b9797f6cd0b08ae362c00ca0f4c2e5564c42a1ab31e7ba9353b80ed"}
Nov 02 13:56:58.312273 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.312241    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj" event={"ID":"2a0f1ebd-01d2-4ed1-914c-1657a788291d","Type":"ContainerStarted","Data":"8745e7199b4497002aa950588361ba5a2ef1b3e3fc2564a4294eb1b2c266a2cd"}
Nov 02 13:56:58.331996 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.331963    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:56:58.340308 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.340253    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj" podStartSLOduration=2.340235388 podStartE2EDuration="2.340235388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:58.338023505 +0000 UTC m=+794.916112371" watchObservedRunningTime="2026-11-02 13:56:58.340235388 +0000 UTC m=+794.918324256"
Nov 02 13:56:58.353239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.353206    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 13:56:58.353375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:58.353270    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 13:56:59.157021 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.156986    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:59.157450 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.157428    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:59.158311 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.158292    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 13:56:59.319181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.319147    5572 generic.go:356] "Generic (PLEG): container finished" podID="2a0f1ebd-01d2-4ed1-914c-1657a788291d" containerID="c37a91ef9b9797f6cd0b08ae362c00ca0f4c2e5564c42a1ab31e7ba9353b80ed" exitCode=0
Nov 02 13:56:59.319364 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.319222    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj" event={"ID":"2a0f1ebd-01d2-4ed1-914c-1657a788291d","Type":"ContainerDied","Data":"c37a91ef9b9797f6cd0b08ae362c00ca0f4c2e5564c42a1ab31e7ba9353b80ed"}
Nov 02 13:56:59.357083 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.357059    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:59.357375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.357357    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:56:59.358148 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:56:59.358126    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 13:57:00.481078 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.481049    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:57:00.616304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.616224    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-xks-ca\") pod \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") "
Nov 02 13:57:00.616304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.616284    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0f1ebd-01d2-4ed1-914c-1657a788291d-kube-api-access-mc9mb\" (UniqueName: \"kubernetes.io/projected/2a0f1ebd-01d2-4ed1-914c-1657a788291d-kube-api-access-mc9mb\") pod \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") "
Nov 02 13:57:00.616490 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.616473    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-openshift-ca\") pod \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\" (UID: \"2a0f1ebd-01d2-4ed1-914c-1657a788291d\") "
Nov 02 13:57:00.616843 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.616817    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-openshift-ca" pod "2a0f1ebd-01d2-4ed1-914c-1657a788291d" (UID: "2a0f1ebd-01d2-4ed1-914c-1657a788291d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:00.617001 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.616980    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:00.618370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.618346    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-xks-ca" pod "2a0f1ebd-01d2-4ed1-914c-1657a788291d" (UID: "2a0f1ebd-01d2-4ed1-914c-1657a788291d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:00.618509 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.618488    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0f1ebd-01d2-4ed1-914c-1657a788291d-kube-api-access-mc9mb" pod "2a0f1ebd-01d2-4ed1-914c-1657a788291d" (UID: "2a0f1ebd-01d2-4ed1-914c-1657a788291d"). InnerVolumeSpecName "kube-api-access-mc9mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:00.718253 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.718226    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0f1ebd-01d2-4ed1-914c-1657a788291d-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:00.718253 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:00.718249    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9mb\" (UniqueName: \"kubernetes.io/projected/2a0f1ebd-01d2-4ed1-914c-1657a788291d-kube-api-access-mc9mb\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:01.333955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:01.333928    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"
Nov 02 13:57:01.333955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:01.333943    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj" event={"ID":"2a0f1ebd-01d2-4ed1-914c-1657a788291d","Type":"ContainerDied","Data":"8745e7199b4497002aa950588361ba5a2ef1b3e3fc2564a4294eb1b2c266a2cd"}
Nov 02 13:57:01.334172 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:01.333987    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8745e7199b4497002aa950588361ba5a2ef1b3e3fc2564a4294eb1b2c266a2cd"
Nov 02 13:57:04.771641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:04.770325    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:57:04.815525 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:04.815497    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:57:05.032827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.032750    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"]
Nov 02 13:57:05.033285 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.033272    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2a0f1ebd-01d2-4ed1-914c-1657a788291d" containerName="revoke-keys"
Nov 02 13:57:05.033332 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.033286    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0f1ebd-01d2-4ed1-914c-1657a788291d" containerName="revoke-keys"
Nov 02 13:57:05.033416 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.033407    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0f1ebd-01d2-4ed1-914c-1657a788291d" containerName="revoke-keys"
Nov 02 13:57:05.037283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.037263    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.043785 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.043764    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:57:05.046454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.046438    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:05.054604 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.053388    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"]
Nov 02 13:57:05.132230 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.132197    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.132424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.132243    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6cvm\" (UniqueName: \"kubernetes.io/projected/efa2232b-97d8-4bc3-8a37-48c3c8043569-kube-api-access-b6cvm\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.132424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.132390    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.234238 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.234195    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.234440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.234256    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6cvm\" (UniqueName: \"kubernetes.io/projected/efa2232b-97d8-4bc3-8a37-48c3c8043569-kube-api-access-b6cvm\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.234440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.234332    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.235033 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.235014    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.236807 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.236779    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.247729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.247699    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6cvm\" (UniqueName: \"kubernetes.io/projected/efa2232b-97d8-4bc3-8a37-48c3c8043569-kube-api-access-b6cvm\") pod \"maas-api-revoke-sub-64a8057c-6txtr\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.351959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.351923    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:05.433038 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.433004    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"]
Nov 02 13:57:05.438993 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.438955    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:57:05.439249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.439232    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.460341 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.460313    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"]
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.537597    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.537647    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.537790    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzbb5\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-kube-api-access-fzbb5\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.537831    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.537864    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.537935    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.537984    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.538069    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.538719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.538167    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.551599 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.551553    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"]
Nov 02 13:57:05.555971 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.555941    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.561605 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.561536    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1eb567\"/\"distinct-w6-1eb567-kserve-self-signed-certs\""
Nov 02 13:57:05.565212 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.565183    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"]
Nov 02 13:57:05.636548 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:05.636518    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa2232b_97d8_4bc3_8a37_48c3c8043569.slice/crio-55d916d01fd7c260f38d738b1a560a88759136e347d2c5bc7aff95cf71cdf415 WatchSource:0}: Error finding container 55d916d01fd7c260f38d738b1a560a88759136e347d2c5bc7aff95cf71cdf415: Status 404 returned error can't find the container with id 55d916d01fd7c260f38d738b1a560a88759136e347d2c5bc7aff95cf71cdf415
Nov 02 13:57:05.637490 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.637461    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"]
Nov 02 13:57:05.638954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.638931    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639083 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.638969    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639083 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639025    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-model-cache\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.639197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639130    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6305498-2daa-47b9-b2a4-e6febb655020-tls-certs\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.639251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639198    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbb5\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-kube-api-access-fzbb5\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639233    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kdt\" (UniqueName: \"kubernetes.io/projected/d6305498-2daa-47b9-b2a4-e6febb655020-kube-api-access-97kdt\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.639356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639260    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639270    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639310    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639332    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-dshm\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.639723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639383    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639414    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639444    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-home\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.639723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639483    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.639723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639535    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-tmp-dir\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.639723 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639644    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.640034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.639917    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.640034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.640003    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.640651 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.640422    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.640651 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.640444    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.641872 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.641846    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.642271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.642250    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.652005 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.651971    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.652185 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.652168    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbb5\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-kube-api-access-fzbb5\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.740754    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6305498-2daa-47b9-b2a4-e6febb655020-tls-certs\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.740829    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kdt\" (UniqueName: \"kubernetes.io/projected/d6305498-2daa-47b9-b2a4-e6febb655020-kube-api-access-97kdt\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.740868    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-dshm\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.740967    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-home\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.741087    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-tmp-dir\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.741149    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-model-cache\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.741646    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-model-cache\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.742675    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-home\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.745690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.743236    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-tmp-dir\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.752316 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.752246    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6305498-2daa-47b9-b2a4-e6febb655020-tls-certs\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.767581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.765016    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-dshm\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.767581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.767475    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:05.772284 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.772207    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kdt\" (UniqueName: \"kubernetes.io/projected/d6305498-2daa-47b9-b2a4-e6febb655020-kube-api-access-97kdt\") pod \"distinct-w6-1eb567-kserve-575547f587-jlnsx\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:05.873610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:05.872442    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:06.105473 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:06.105442    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b87a981_ea31_4e9d_a355_eb7f9ba33924.slice/crio-4228d0d1c29622e9f73720f2c253a501f1547e01b7db45bb70bdcd9e94f1b6b7 WatchSource:0}: Error finding container 4228d0d1c29622e9f73720f2c253a501f1547e01b7db45bb70bdcd9e94f1b6b7: Status 404 returned error can't find the container with id 4228d0d1c29622e9f73720f2c253a501f1547e01b7db45bb70bdcd9e94f1b6b7
Nov 02 13:57:06.105789 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.105764    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"]
Nov 02 13:57:06.108178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.108149    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:06.108258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.108212    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:06.108258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.108240    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:06.173333 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.173304    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"]
Nov 02 13:57:06.173604 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:06.173543    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6305498_2daa_47b9_b2a4_e6febb655020.slice/crio-749b82b2548378fdf2167f6fff47abbf4469537248b97a506790aa7b4b7ea35a WatchSource:0}: Error finding container 749b82b2548378fdf2167f6fff47abbf4469537248b97a506790aa7b4b7ea35a: Status 404 returned error can't find the container with id 749b82b2548378fdf2167f6fff47abbf4469537248b97a506790aa7b4b7ea35a
Nov 02 13:57:06.509593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.507289    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" event={"ID":"3b87a981-ea31-4e9d-a355-eb7f9ba33924","Type":"ContainerStarted","Data":"7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a"}
Nov 02 13:57:06.509593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.507359    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" event={"ID":"3b87a981-ea31-4e9d-a355-eb7f9ba33924","Type":"ContainerStarted","Data":"4228d0d1c29622e9f73720f2c253a501f1547e01b7db45bb70bdcd9e94f1b6b7"}
Nov 02 13:57:06.511261 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.510119    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr" event={"ID":"efa2232b-97d8-4bc3-8a37-48c3c8043569","Type":"ContainerStarted","Data":"00dd08e5d2114c389e19cc6874aebf5d8f69a78e89569754c0c1d6df547e8318"}
Nov 02 13:57:06.511261 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.510185    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr" event={"ID":"efa2232b-97d8-4bc3-8a37-48c3c8043569","Type":"ContainerStarted","Data":"55d916d01fd7c260f38d738b1a560a88759136e347d2c5bc7aff95cf71cdf415"}
Nov 02 13:57:06.516300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.516255    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" event={"ID":"d6305498-2daa-47b9-b2a4-e6febb655020","Type":"ContainerStarted","Data":"5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec"}
Nov 02 13:57:06.516796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.516774    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" event={"ID":"d6305498-2daa-47b9-b2a4-e6febb655020","Type":"ContainerStarted","Data":"749b82b2548378fdf2167f6fff47abbf4469537248b97a506790aa7b4b7ea35a"}
Nov 02 13:57:06.534663 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.534612    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr" podStartSLOduration=1.5345946449999999 podStartE2EDuration="1.534594645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:06.532057911 +0000 UTC m=+803.110146779" watchObservedRunningTime="2026-11-02 13:57:06.534594645 +0000 UTC m=+803.112683511"
Nov 02 13:57:06.557058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.556983    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podStartSLOduration=1.556961995 podStartE2EDuration="1.556961995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:06.554784492 +0000 UTC m=+803.132873362" watchObservedRunningTime="2026-11-02 13:57:06.556961995 +0000 UTC m=+803.135050865"
Nov 02 13:57:06.589092 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.589006    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podStartSLOduration=1.5889827109999999 podStartE2EDuration="1.588982711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:06.586306751 +0000 UTC m=+803.164395625" watchObservedRunningTime="2026-11-02 13:57:06.588982711 +0000 UTC m=+803.167071577"
Nov 02 13:57:06.769887 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.769805    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:06.876352 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:06.876315    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:07.547981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:07.547950    5572 generic.go:356] "Generic (PLEG): container finished" podID="efa2232b-97d8-4bc3-8a37-48c3c8043569" containerID="00dd08e5d2114c389e19cc6874aebf5d8f69a78e89569754c0c1d6df547e8318" exitCode=0
Nov 02 13:57:07.550955 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:07.550929    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr" event={"ID":"efa2232b-97d8-4bc3-8a37-48c3c8043569","Type":"ContainerDied","Data":"00dd08e5d2114c389e19cc6874aebf5d8f69a78e89569754c0c1d6df547e8318"}
Nov 02 13:57:07.769926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:07.769890    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:57:07.770099 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:07.769955    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 13:57:08.768459 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.768425    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:57:08.768856 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.768490    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 13:57:08.813326 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.813297    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:08.920673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.920588    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa2232b-97d8-4bc3-8a37-48c3c8043569-kube-api-access-b6cvm\" (UniqueName: \"kubernetes.io/projected/efa2232b-97d8-4bc3-8a37-48c3c8043569-kube-api-access-b6cvm\") pod \"efa2232b-97d8-4bc3-8a37-48c3c8043569\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") "
Nov 02 13:57:08.920827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.920723    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-xks-ca\") pod \"efa2232b-97d8-4bc3-8a37-48c3c8043569\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") "
Nov 02 13:57:08.920827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.920754    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-openshift-ca\") pod \"efa2232b-97d8-4bc3-8a37-48c3c8043569\" (UID: \"efa2232b-97d8-4bc3-8a37-48c3c8043569\") "
Nov 02 13:57:08.921201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.921169    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-openshift-ca" pod "efa2232b-97d8-4bc3-8a37-48c3c8043569" (UID: "efa2232b-97d8-4bc3-8a37-48c3c8043569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:08.921368 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.921353    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:08.923022 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.922997    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa2232b-97d8-4bc3-8a37-48c3c8043569-kube-api-access-b6cvm" pod "efa2232b-97d8-4bc3-8a37-48c3c8043569" (UID: "efa2232b-97d8-4bc3-8a37-48c3c8043569"). InnerVolumeSpecName "kube-api-access-b6cvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:08.923022 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:08.923006    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-xks-ca" pod "efa2232b-97d8-4bc3-8a37-48c3c8043569" (UID: "efa2232b-97d8-4bc3-8a37-48c3c8043569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:09.023169 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:09.023132    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6cvm\" (UniqueName: \"kubernetes.io/projected/efa2232b-97d8-4bc3-8a37-48c3c8043569-kube-api-access-b6cvm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:09.023169 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:09.023163    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa2232b-97d8-4bc3-8a37-48c3c8043569-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:09.611264 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:09.611229    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr" event={"ID":"efa2232b-97d8-4bc3-8a37-48c3c8043569","Type":"ContainerDied","Data":"55d916d01fd7c260f38d738b1a560a88759136e347d2c5bc7aff95cf71cdf415"}
Nov 02 13:57:09.611470 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:09.611281    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d916d01fd7c260f38d738b1a560a88759136e347d2c5bc7aff95cf71cdf415"
Nov 02 13:57:09.611470 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:09.611332    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"
Nov 02 13:57:09.768622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:09.768591    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:57:09.769028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:09.768645    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 13:57:10.768196 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:10.768164    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:57:10.768371 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:10.768222    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 13:57:11.768858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:11.768821    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:57:11.769934 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:11.769421    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 13:57:12.534400 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.534362    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"]
Nov 02 13:57:12.534990 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.534975    5572 cpu_manager.go:414] "containerMap: removing container" podUID="efa2232b-97d8-4bc3-8a37-48c3c8043569" containerName="revoke-keys"
Nov 02 13:57:12.535054 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.534991    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa2232b-97d8-4bc3-8a37-48c3c8043569" containerName="revoke-keys"
Nov 02 13:57:12.535125 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.535116    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa2232b-97d8-4bc3-8a37-48c3c8043569" containerName="revoke-keys"
Nov 02 13:57:12.539834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.539815    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.544544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.544516    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:12.544699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.544622    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:57:12.547455 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.547422    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"]
Nov 02 13:57:12.671297 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.671261    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.671514 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.671318    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdqb\" (UniqueName: \"kubernetes.io/projected/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-kube-api-access-lbdqb\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.671626 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.671593    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.768719 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.768682    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:57:12.768888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.768748    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 13:57:12.772534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.772488    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.772757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.772621    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.772757 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.772674    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdqb\" (UniqueName: \"kubernetes.io/projected/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-kube-api-access-lbdqb\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.773603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.773539    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.775320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.775271    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.786145 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.786070    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdqb\" (UniqueName: \"kubernetes.io/projected/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-kube-api-access-lbdqb\") pod \"maas-api-revoke-sub-662b0641-jqx57\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:12.855046 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:12.854871    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:13.152677 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:13.152573    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1656d28_5b08_4f2f_ba3d_8008c6c103bf.slice/crio-08ccb698bb20007528eb71b71f9c789560447eb4524250f76137224416d573f3 WatchSource:0}: Error finding container 08ccb698bb20007528eb71b71f9c789560447eb4524250f76137224416d573f3: Status 404 returned error can't find the container with id 08ccb698bb20007528eb71b71f9c789560447eb4524250f76137224416d573f3
Nov 02 13:57:13.153204 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:13.153161    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"]
Nov 02 13:57:13.657163 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:13.657123    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57" event={"ID":"c1656d28-5b08-4f2f-ba3d-8008c6c103bf","Type":"ContainerStarted","Data":"bcd45432acc2042531613f19037abb4a8713fcd9e5ce75efc4f525adee5407b1"}
Nov 02 13:57:13.657390 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:13.657188    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57" event={"ID":"c1656d28-5b08-4f2f-ba3d-8008c6c103bf","Type":"ContainerStarted","Data":"08ccb698bb20007528eb71b71f9c789560447eb4524250f76137224416d573f3"}
Nov 02 13:57:13.677345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:13.677294    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57" podStartSLOduration=1.6772799489999999 podStartE2EDuration="1.677279949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:13.674650757 +0000 UTC m=+810.252739636" watchObservedRunningTime="2026-11-02 13:57:13.677279949 +0000 UTC m=+810.255368816"
Nov 02 13:57:13.773026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:13.772942    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:13.773447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:13.773428    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:13.774249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:13.774226    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 13:57:14.666382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:14.666351    5572 generic.go:356] "Generic (PLEG): container finished" podID="c1656d28-5b08-4f2f-ba3d-8008c6c103bf" containerID="bcd45432acc2042531613f19037abb4a8713fcd9e5ce75efc4f525adee5407b1" exitCode=0
Nov 02 13:57:14.666630 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:14.666440    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57" event={"ID":"c1656d28-5b08-4f2f-ba3d-8008c6c103bf","Type":"ContainerDied","Data":"bcd45432acc2042531613f19037abb4a8713fcd9e5ce75efc4f525adee5407b1"}
Nov 02 13:57:15.848202 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.848176    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:15.875742 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.875706    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:15.889630 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.889548    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:15.936777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.936742    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-openshift-ca\") pod \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") "
Nov 02 13:57:15.936947 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.936839    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-kube-api-access-lbdqb\" (UniqueName: \"kubernetes.io/projected/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-kube-api-access-lbdqb\") pod \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") "
Nov 02 13:57:15.936947 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.936940    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-xks-ca\") pod \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\" (UID: \"c1656d28-5b08-4f2f-ba3d-8008c6c103bf\") "
Nov 02 13:57:15.937178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.937156    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-openshift-ca" pod "c1656d28-5b08-4f2f-ba3d-8008c6c103bf" (UID: "c1656d28-5b08-4f2f-ba3d-8008c6c103bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:15.937471 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.937455    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:15.939207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.939158    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-xks-ca" pod "c1656d28-5b08-4f2f-ba3d-8008c6c103bf" (UID: "c1656d28-5b08-4f2f-ba3d-8008c6c103bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:15.939269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:15.939253    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-kube-api-access-lbdqb" pod "c1656d28-5b08-4f2f-ba3d-8008c6c103bf" (UID: "c1656d28-5b08-4f2f-ba3d-8008c6c103bf"). InnerVolumeSpecName "kube-api-access-lbdqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:16.039091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.039049    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdqb\" (UniqueName: \"kubernetes.io/projected/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-kube-api-access-lbdqb\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:16.039091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.039082    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1656d28-5b08-4f2f-ba3d-8008c6c103bf-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:16.695999 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.695969    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"
Nov 02 13:57:16.696181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.696010    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57" event={"ID":"c1656d28-5b08-4f2f-ba3d-8008c6c103bf","Type":"ContainerDied","Data":"08ccb698bb20007528eb71b71f9c789560447eb4524250f76137224416d573f3"}
Nov 02 13:57:16.696181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.696063    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08ccb698bb20007528eb71b71f9c789560447eb4524250f76137224416d573f3"
Nov 02 13:57:16.710753 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.710724    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:16.718239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.718205    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"]
Nov 02 13:57:16.719102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.719083    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c1656d28-5b08-4f2f-ba3d-8008c6c103bf" containerName="revoke-keys"
Nov 02 13:57:16.719102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.719104    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1656d28-5b08-4f2f-ba3d-8008c6c103bf" containerName="revoke-keys"
Nov 02 13:57:16.719283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.719227    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1656d28-5b08-4f2f-ba3d-8008c6c103bf" containerName="revoke-keys"
Nov 02 13:57:16.723418 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.723395    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.738931 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.737512    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4fc49b\"/\"distinct-2-w3-4fc49b-kserve-self-signed-certs\""
Nov 02 13:57:16.742602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.742578    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"]
Nov 02 13:57:16.857679 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.857646    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-dshm\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.858073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.857697    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-home\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.858073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.857717    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-tmp-dir\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.858073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.857757    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13cc652-6d83-44e5-bcb0-77481d846a54-tls-certs\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.858073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.857894    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnbg\" (UniqueName: \"kubernetes.io/projected/e13cc652-6d83-44e5-bcb0-77481d846a54-kube-api-access-hbnbg\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.858073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.857977    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-model-cache\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.961354    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-dshm\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.961433    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-home\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.961461    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-tmp-dir\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.961527    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13cc652-6d83-44e5-bcb0-77481d846a54-tls-certs\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.961596    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnbg\" (UniqueName: \"kubernetes.io/projected/e13cc652-6d83-44e5-bcb0-77481d846a54-kube-api-access-hbnbg\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.961648    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-model-cache\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.962300    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-model-cache\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.962889    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-tmp-dir\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.965641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.963404    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-home\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.981656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.970954    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-dshm\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:16.998329 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:16.998087    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13cc652-6d83-44e5-bcb0-77481d846a54-tls-certs\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:17.012288 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:17.007720    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnbg\" (UniqueName: \"kubernetes.io/projected/e13cc652-6d83-44e5-bcb0-77481d846a54-kube-api-access-hbnbg\") pod \"distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") " pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:17.044197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:17.044165    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:17.335540 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:17.335510    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"]
Nov 02 13:57:17.340454 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:17.340422    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode13cc652_6d83_44e5_bcb0_77481d846a54.slice/crio-1f778eac2ef5c720b975b601127f9a150d1bf72549af4db072b9a61fc9504919 WatchSource:0}: Error finding container 1f778eac2ef5c720b975b601127f9a150d1bf72549af4db072b9a61fc9504919: Status 404 returned error can't find the container with id 1f778eac2ef5c720b975b601127f9a150d1bf72549af4db072b9a61fc9504919
Nov 02 13:57:17.715981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:17.715873    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" event={"ID":"e13cc652-6d83-44e5-bcb0-77481d846a54","Type":"ContainerStarted","Data":"09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08"}
Nov 02 13:57:17.715981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:17.715939    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" event={"ID":"e13cc652-6d83-44e5-bcb0-77481d846a54","Type":"ContainerStarted","Data":"1f778eac2ef5c720b975b601127f9a150d1bf72549af4db072b9a61fc9504919"}
Nov 02 13:57:17.738440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:17.738378    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podStartSLOduration=1.7383648489999999 podStartE2EDuration="1.738364849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:17.735176499 +0000 UTC m=+814.313265365" watchObservedRunningTime="2026-11-02 13:57:17.738364849 +0000 UTC m=+814.316453754"
Nov 02 13:57:18.044619 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:18.044530    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:20.216899 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.216864    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"]
Nov 02 13:57:20.221314 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.221298    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.237726 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.237697    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"]
Nov 02 13:57:20.273722 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.273688    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"]
Nov 02 13:57:20.277840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.277821    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.285729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.284903    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"]
Nov 02 13:57:20.348655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.348617    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.348841 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.348674    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.348841 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.348773    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.348941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.348843    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6ww\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-kube-api-access-6q6ww\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.348941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.348888    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.348941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.348918    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.349067 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.349053    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.349104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.349078    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.349156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.349130    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.450608 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450555    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.450802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450631    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.450802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450658    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.450802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450686    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.450802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450713    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.450994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450817    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.450994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450857    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168158b6-2594-43f2-916b-9bf50573cb81-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.450994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450914    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.450994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.450969    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6ww\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-kube-api-access-6q6ww\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451019    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451048    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451089    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451121    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.451180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451156    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.451419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451181    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168158b6-2594-43f2-916b-9bf50573cb81-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.451419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451226    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.451419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451279    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frrk\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-kube-api-access-8frrk\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.451419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451305    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451411    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451429    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451469    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451842 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451687    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.451966 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.451945    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.453352 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.453324    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.453706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.453689    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.463664 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.463638    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6ww\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-kube-api-access-6q6ww\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.463811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.463726    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.537963 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.537886    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:20.560830 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.560788    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168158b6-2594-43f2-916b-9bf50573cb81-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.560969 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.560871    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.560976    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561001    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561092 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561029    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168158b6-2594-43f2-916b-9bf50573cb81-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561092 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561071    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561121    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frrk\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-kube-api-access-8frrk\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561620    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-certs\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561750 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561630    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561750 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561709    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561750 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561702    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.561897 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.561862    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-credential-socket\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.562065 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.562041    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-data\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.562967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.562943    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168158b6-2594-43f2-916b-9bf50573cb81-istiod-ca-cert\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.564700 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.564677    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-envoy\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.565091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.565070    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168158b6-2594-43f2-916b-9bf50573cb81-istio-podinfo\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.573919 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.573895    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"]
Nov 02 13:57:20.579603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.579461    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.579915 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.579890    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-istio-token\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.580033 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.580015    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frrk\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-kube-api-access-8frrk\") pod \"e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") " pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.597599 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.597537    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"]
Nov 02 13:57:20.599417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.599392    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:20.665042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665005    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665088    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665157    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665186    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665221    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665478 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665312    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mb87\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-kube-api-access-2mb87\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665478 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665366    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665478 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665444    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.665665 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.665487    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767281 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767238    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767481 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767324    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767481 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767394    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767481 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767465    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767718 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767537    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767718 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767587    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767718 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767619    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767718    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mb87\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-kube-api-access-2mb87\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.767869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767777    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.768002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.767972    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.768228 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.768209    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.768325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.768301    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.768462 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.768443    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.768869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.768846    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.770615 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.770598    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.770879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.770858    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.785249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.785210    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.785367 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.785276    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mb87\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-kube-api-access-2mb87\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.823744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.823715    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"]
Nov 02 13:57:20.829160 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.829137    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.835967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.835941    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4fc49b-openshift-default-dockercfg-mnqnf\""
Nov 02 13:57:20.841633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.841609    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"]
Nov 02 13:57:20.857733 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.857691    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:20.857868 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.857787    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:20.857868 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.857831    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.867842    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"]
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870283    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870332    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870443    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870474    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870505    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-478gx\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-kube-api-access-478gx\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870576    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870612    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870690    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.872597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.870720    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.974805    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.974855    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.974969    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.974996    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.975021    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478gx\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-kube-api-access-478gx\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.975075    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.975106    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.975187    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.975211    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.976041    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.976793    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.977019    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.977469    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.977750    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:20.984922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.983081    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:20.994596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.986952    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"]
Nov 02 13:57:20.994596 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:20.991147    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168158b6_2594_43f2_916b_9bf50573cb81.slice/crio-1fa3a528c254bbfb8e30292f82898f3393ea66d11f98814a277f358eb4dafff3 WatchSource:0}: Error finding container 1fa3a528c254bbfb8e30292f82898f3393ea66d11f98814a277f358eb4dafff3: Status 404 returned error can't find the container with id 1fa3a528c254bbfb8e30292f82898f3393ea66d11f98814a277f358eb4dafff3
Nov 02 13:57:21.002623 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:20.996074    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:21.003779 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.003731    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.003947 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.003836    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.003947 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.003891    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.004978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.004951    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:21.005714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.005692    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:21.007645 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.007619    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-478gx\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-kube-api-access-478gx\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:21.146393 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.146363    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:21.312943 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:21.312851    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5a41fc_f27f_4ba1_9ca9_617fd1f1039c.slice/crio-614a1bfe1dc7b9935dd4a601124976ea74576980e6436dd1429d06df56c5a2c7 WatchSource:0}: Error finding container 614a1bfe1dc7b9935dd4a601124976ea74576980e6436dd1429d06df56c5a2c7: Status 404 returned error can't find the container with id 614a1bfe1dc7b9935dd4a601124976ea74576980e6436dd1429d06df56c5a2c7
Nov 02 13:57:21.319091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.317988    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.319091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.318073    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.319091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.318113    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.319091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.318167    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"]
Nov 02 13:57:21.451453 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:57:21.451360    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:57:21.486370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.484949    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"]
Nov 02 13:57:21.501295 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.500846    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.501295 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.500949    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.501295 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.500995    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:21.800673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.800304    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" event={"ID":"168158b6-2594-43f2-916b-9bf50573cb81","Type":"ContainerStarted","Data":"b75ef31780f150e1f298d99e76bc2db327d8eb68788929acc07b573a6b6ab6b9"}
Nov 02 13:57:21.800673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.800390    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" event={"ID":"168158b6-2594-43f2-916b-9bf50573cb81","Type":"ContainerStarted","Data":"1fa3a528c254bbfb8e30292f82898f3393ea66d11f98814a277f358eb4dafff3"}
Nov 02 13:57:21.823596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.819459    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" event={"ID":"3eeee583-d5dc-42eb-84a0-a5aa32d80498","Type":"ContainerStarted","Data":"79c2b2385953ab955d7bd33146064a831c39aaa246eac1c09e1798336dfc06e9"}
Nov 02 13:57:21.823596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.819528    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" event={"ID":"3eeee583-d5dc-42eb-84a0-a5aa32d80498","Type":"ContainerStarted","Data":"cc7a1e033912c80c51c62464dd6c98305709db2108a3ae2aa13e49682575a8bd"}
Nov 02 13:57:21.916603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.898851    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" event={"ID":"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c","Type":"ContainerStarted","Data":"b88fe6a51005f0e04a249e3546ed40d68ae33ea5ebd8aee1ad91feeaeefea359"}
Nov 02 13:57:21.916603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.898922    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" event={"ID":"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c","Type":"ContainerStarted","Data":"614a1bfe1dc7b9935dd4a601124976ea74576980e6436dd1429d06df56c5a2c7"}
Nov 02 13:57:21.952642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.943589    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podStartSLOduration=1.943549728 podStartE2EDuration="1.943549728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:21.868742727 +0000 UTC m=+818.446831600" watchObservedRunningTime="2026-11-02 13:57:21.943549728 +0000 UTC m=+818.521638595"
Nov 02 13:57:21.978300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.975720    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" event={"ID":"92181eed-6e76-499d-8364-fd0e2dd7aa3c","Type":"ContainerStarted","Data":"782ebde05f5d121258e9507aa603563a2706e9eb0bd106f8bdc369d4fd9c7e8a"}
Nov 02 13:57:21.978300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.975791    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" event={"ID":"92181eed-6e76-499d-8364-fd0e2dd7aa3c","Type":"ContainerStarted","Data":"7c1142a6ee57d477a999fdb199cce14ad032d0d6c92e14c1ed0d9c6b5ba880ae"}
Nov 02 13:57:21.989751 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.989651    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:21.991038 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:21.990981    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podStartSLOduration=1.990960699 podStartE2EDuration="1.990960699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:21.941754021 +0000 UTC m=+818.519842889" watchObservedRunningTime="2026-11-02 13:57:21.990960699 +0000 UTC m=+818.569049568"
Nov 02 13:57:22.022584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.000289    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podStartSLOduration=2.00027298 podStartE2EDuration="2.00027298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:21.973121512 +0000 UTC m=+818.551210379" watchObservedRunningTime="2026-11-02 13:57:22.00027298 +0000 UTC m=+818.578361865"
Nov 02 13:57:22.050136 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.050081    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podStartSLOduration=2.050064535 podStartE2EDuration="2.050064535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:22.028214791 +0000 UTC m=+818.606303648" watchObservedRunningTime="2026-11-02 13:57:22.050064535 +0000 UTC m=+818.628153401"
Nov 02 13:57:22.146771 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.146727    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:22.148797 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.148752    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:22.148952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.148800    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:22.539653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.539495    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:22.544603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.541273    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:22.544603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.541323    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:22.602095 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.602053    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:22.613791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.613739    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:22.614075 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.614033    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:23.008640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.999654    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:23.008640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:22.999719    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:23.147587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.146998    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:23.147587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.147059    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:23.490621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.486902    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:57:23.552591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.551711    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:23.552591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.552473    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:23.610072 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.610009    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:23.610288 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.610075    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:23.634587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.632660    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:23.701620 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.694873    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:23.999590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.990225    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:23.999590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:23.990286    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:24.065596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.062909    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:24.152593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.150043    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:24.152593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.150199    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:24.466588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.454375    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:57:24.466588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.457123    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:57:24.472583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.472517    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:24.494587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.492880    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:57:24.494587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.493668    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:57:24.538586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.538358    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:24.538586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.538418    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:24.613042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.611409    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:24.613042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.611470    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:24.671987 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.669656    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:24.671987 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.670272    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:57:24.702621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.695733    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:24.702621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.696146    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:57:24.727642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.723811    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:24.793593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.742756    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:57:24.995159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.995063    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:24.995430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:24.995397    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:25.081213 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.078828    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.081213 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.080013    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:57:25.104800 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.092707    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.166337 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.166299    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:25.166902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.166865    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:25.221048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.221004    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.466837 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.466701    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.468989 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.468950    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:57:25.475751 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.475640    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.476071 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.476052    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:57:25.506607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.506547    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.615230 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.615182    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:25.623197 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.623139    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:25.773956 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.661700    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:25.774218 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.774186    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:25.780958 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.753246    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:57:25.782826 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.753311    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.784010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.674834    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.799622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.778638    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:25.849788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.849753    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:57:25.850038 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.850022    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:57:25.987041 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.987009    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:25.993607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:25.987282    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:26.084330 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.084285    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:57:26.189776 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.189737    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:26.190770 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.190736    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:26.334641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.328667    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.563085 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.559116    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:26.563085 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.559180    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:26.564750 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.564706    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.564968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.564945    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:57:26.568590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.565110    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:57:26.605936 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.601715    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.744662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.744214    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:26.744662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.744275    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:26.744662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.744358    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.787830 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.787786    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.788550 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.788519    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.788807 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.787713    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:57:26.789159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.789128    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:57:26.943635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.932871    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.943635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.934027    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:26.996849 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.996813    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:26.997107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:26.997078    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:27.081597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.075323    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:27.158596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.142714    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:27.158596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.150973    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:27.158596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.151035    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:27.333593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.329769    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:27.333593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.330588    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:57:27.637652 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.636713    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:27.637652 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.636785    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:27.662584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.641392    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:27.662584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.641446    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:27.662584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.655403    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:57:27.684922 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.684875    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:27.718586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.713803    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:27.718586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.714077    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:57:27.718586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.714340    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:57:27.759243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.759206    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 13:57:27.788516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.788477    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:27.803586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.789132    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:57:27.805627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:27.804039    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.013666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.001131    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:28.065946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.065876    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:28.091412 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.070886    5572 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:28.247859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.247798    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.315655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.266648    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:57:28.556786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.271275    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:28.556786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.511855    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:28.556786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.435000    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.556786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.538523    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:28.556786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.538597    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:28.600274 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.600178    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:28.600594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.600546    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:28.655714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.655649    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:28.707284 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.706862    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:28.755584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.748913    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:28.755584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.749443    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.755584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.749832    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 13:57:28.755584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.749917    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.755584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.750221    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.755584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.750859    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:57:28.807896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.807846    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.808842 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.808809    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:57:28.827585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:28.814448    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:57:29.027648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.023479    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:29.027648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.023538    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:29.155588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.154940    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:29.175591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.158688    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:29.248752 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.248699    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.495301 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.495256    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.538684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.538046    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.539180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.538978    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:29.539180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.539023    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:29.600986 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.600948    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:29.601248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.601219    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:29.667219 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.667177    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.716586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.711747    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.752847 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.752690    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.754013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.753493    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:29.758677 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.758645    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.783408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.753590    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:57:29.783837 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.753617    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.784005 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.753534    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.784676 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.784653    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:29.823404 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.823359    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.951586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.875231    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:57:29.951762 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.878773    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:29.951980 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.950736    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:29.951980 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:29.951922    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.011835 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.011748    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:30.012113 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.012069    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:30.171081 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.171049    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:30.171396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.171349    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:30.199476 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.199441    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:30.237043 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.236882    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.252721 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.252671    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.324954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.324916    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:57:30.416753 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.416709    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:30.441707 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.441657    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.487764 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.487711    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.559765 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.559705    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:57:30.585698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.563372    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:30.596587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.585953    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:30.678584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.649922    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:30.678965 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.678846    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:30.693773 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.693727    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.723988 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.723943    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.766519 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.766480    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:57:30.787370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.787330    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:57:30.839645 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.815722    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:30.956714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.956668    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:57:31.016617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.956985    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:57:31.016617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:30.957018    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:57:31.094640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.094095    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:31.094640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.094156    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:31.150587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.147944    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:31.150587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.148009    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:31.257778 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.257714    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:31.562741 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.562697    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:57:31.567141 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.567097    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:31.567435 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.567394    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:31.601005 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.600960    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:31.601286 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.601258    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:31.695755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.695701    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:31.725697 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.725646    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:31.772934 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.771704    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:31.793375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.793324    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:57:31.864644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.863804    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:31.888588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.877887    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:57:31.994586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.987980    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 13:57:31.994586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:31.988056    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 13:57:32.012610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.007954    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:57:32.012610 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.008787    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:57:32.020590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.020517    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:32.213172 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.156322    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:32.213172 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.156385    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:32.213172 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.178928    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:57:32.472639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.471963    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:57:32.506644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.503264    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:57:32.517951 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.510125    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:57:32.547952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.544963    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:57:32.547952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.547230    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:32.547952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.547288    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:32.590825 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.590778    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:32.594375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.594349    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:32.626586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.622901    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:57:32.647331 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.647285    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:57:32.647795 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.647679    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 13:57:32.714398 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.712580    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:57:32.715103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.714867    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:57:32.716908 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:32.716869    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 02 13:57:33.154587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.151708    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 13:57:33.154587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.151771    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 13:57:33.585488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.585451    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 13:57:33.585832 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.585799    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 13:57:33.866364 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.865802    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:33.866794 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.866774    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:33.870061 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.870035    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 13:57:33.992325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.992290    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:33.992612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:33.992545    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:34.564391 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:34.564359    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:34.566382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:34.566361    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:34.574928 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:34.574896    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 13:57:34.581929 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:34.581900    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:34.583454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:34.583431    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:34.611017 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:34.603387    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 13:57:35.188815 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:35.188768    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:35.189004 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:35.188827    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:35.363275 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:35.363236    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:35.363699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:35.363678    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:35.364453 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:35.364435    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 13:57:37.058088 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:37.058059    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:37.069647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:37.069623    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:48.418474 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.418439    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"]
Nov 02 13:57:48.424362 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.424341    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.431613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.429728    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:48.431613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.429817    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:57:48.435168 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.435145    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"]
Nov 02 13:57:48.459690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.459657    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.459858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.459773    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmjt\" (UniqueName: \"kubernetes.io/projected/e3905545-6743-4511-9d4c-8fc0bacaafdc-kube-api-access-wvmjt\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.460011 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.459991    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.561260 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.561234    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.561447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.561294    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.561447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.561337    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmjt\" (UniqueName: \"kubernetes.io/projected/e3905545-6743-4511-9d4c-8fc0bacaafdc-kube-api-access-wvmjt\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.561961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.561944    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.563977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.563955    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.576648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.576619    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmjt\" (UniqueName: \"kubernetes.io/projected/e3905545-6743-4511-9d4c-8fc0bacaafdc-kube-api-access-wvmjt\") pod \"maas-api-revoke-sub-d64ed6a7-8n5mw\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:48.741885 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:48.741804    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:49.059225 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:49.059193    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3905545_6743_4511_9d4c_8fc0bacaafdc.slice/crio-ec9cc6fbfd376199dd82aeb87b7a33706f93138951029c573c8287e1b29f151a WatchSource:0}: Error finding container ec9cc6fbfd376199dd82aeb87b7a33706f93138951029c573c8287e1b29f151a: Status 404 returned error can't find the container with id ec9cc6fbfd376199dd82aeb87b7a33706f93138951029c573c8287e1b29f151a
Nov 02 13:57:49.064068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:49.064035    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"]
Nov 02 13:57:49.531844 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:49.531532    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw" event={"ID":"e3905545-6743-4511-9d4c-8fc0bacaafdc","Type":"ContainerStarted","Data":"34085923cb928a13c1e4bf1399983674c893f2b7342e506ad422b89bff978b14"}
Nov 02 13:57:49.531844 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:49.531675    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw" event={"ID":"e3905545-6743-4511-9d4c-8fc0bacaafdc","Type":"ContainerStarted","Data":"ec9cc6fbfd376199dd82aeb87b7a33706f93138951029c573c8287e1b29f151a"}
Nov 02 13:57:49.576238 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:49.576189    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw" podStartSLOduration=1.576171057 podStartE2EDuration="1.576171057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:49.57209763 +0000 UTC m=+846.150186498" watchObservedRunningTime="2026-11-02 13:57:49.576171057 +0000 UTC m=+846.154259925"
Nov 02 13:57:50.540748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.540712    5572 generic.go:356] "Generic (PLEG): container finished" podID="e3905545-6743-4511-9d4c-8fc0bacaafdc" containerID="34085923cb928a13c1e4bf1399983674c893f2b7342e506ad422b89bff978b14" exitCode=0
Nov 02 13:57:50.541138 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.540803    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw" event={"ID":"e3905545-6743-4511-9d4c-8fc0bacaafdc","Type":"ContainerDied","Data":"34085923cb928a13c1e4bf1399983674c893f2b7342e506ad422b89bff978b14"}
Nov 02 13:57:50.624248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.624218    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"]
Nov 02 13:57:50.628773 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.628741    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.666313 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.666281    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"]
Nov 02 13:57:50.706150 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706115    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706301    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706327    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706345    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706495 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706365    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706601    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706652    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706714    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nncv\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-kube-api-access-4nncv\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.706847 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.706831    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.746694 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.746659    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"]
Nov 02 13:57:50.751218 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.751196    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.775014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.774983    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"]
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.808837    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlbs4\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-kube-api-access-rlbs4\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809486    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809553    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809627    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809666    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809724    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809760    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809845    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809890    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809917    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.809993    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810027    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810087    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810140    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810175    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810229    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810275    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nncv\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-kube-api-access-4nncv\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810308    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421a53e0-3b6f-42a5-8256-71e356f87c49-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.810935    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-credential-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.811632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.811253    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-certs\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.813821 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.813796    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istiod-ca-cert\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.816481 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.814834    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-socket\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.816774 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.814543    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-data\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.859127 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.848492    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-podinfo\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.922841    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.922963    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.923031    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.923155    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421a53e0-3b6f-42a5-8256-71e356f87c49-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.923216    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlbs4\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-kube-api-access-rlbs4\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.923254    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.923316    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.923365    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.925688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.923405    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.929013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.928954    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.929158 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.929008    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.929158 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.928312    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.935876 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.935837    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nncv\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-kube-api-access-4nncv\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.936262 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.936238    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.936397 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.936357    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-token\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.936397 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.936382    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.936696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.936671    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.937415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.937392    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421a53e0-3b6f-42a5-8256-71e356f87c49-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.939164 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.939138    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-envoy\") pod \"e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") " pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.943081 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.942318    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"]
Nov 02 13:57:50.948745 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.948730    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:50.949806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.949792    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:50.963505 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.963483    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"]
Nov 02 13:57:50.970041 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.970015    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:50.970480 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.970461    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:50.985710 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.985686    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"]
Nov 02 13:57:50.991804 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:50.991644    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlbs4\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-kube-api-access-rlbs4\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:51.028438 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.028404    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"]
Nov 02 13:57:51.034763 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.034730    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.035063 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.034998    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.035284 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.035264    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnw9\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-kube-api-access-cnnw9\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.035437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.035422    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.035546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.035531    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.035678 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.035661    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.035777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.035764    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.035898 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.035881    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.036065 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.036041    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.036314 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.036295    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.036508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.036492    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.036735 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.036716    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe76bd4-674d-44a9-923f-2e473cb6e182-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.036900 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.036884    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.037044 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.037025    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.037170 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.037153    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2qd\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-kube-api-access-8d2qd\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.037384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.037365    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.037512 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.037497    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.037662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.037647    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.108583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.076109    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.215797    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe76bd4-674d-44a9-923f-2e473cb6e182-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.215865    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.215903    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.215939    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2qd\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-kube-api-access-8d2qd\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.215987    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216016    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216047    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216149    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216187    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216218    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnw9\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-kube-api-access-cnnw9\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216273    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216298    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216322    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216352    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216397    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216431    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216485    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.216599    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.217070    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.217785    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe76bd4-674d-44a9-923f-2e473cb6e182-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.219147    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.220059 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.219625    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.221378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.218480    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.221378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.220580    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.221378 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.220851    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.242299 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.227015    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.305590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.257532    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.319643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.292253    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.319643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.295114    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.326644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.320511    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnw9\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-kube-api-access-cnnw9\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.326644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.326417    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.334584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.327059    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2qd\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-kube-api-access-8d2qd\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.334584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.327985    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.334584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.333003    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.355105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.348599    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"]
Nov 02 13:57:51.360882 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.358471    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.369642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.369414    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"]
Nov 02 13:57:51.413338 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.392144    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.516907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.496785    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625017    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b772da3e-1119-4346-aa14-d376a1f137df-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625235    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625351    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625413    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b772da3e-1119-4346-aa14-d376a1f137df-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625598    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625672    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625731    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.625931    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.626132    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5nj\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-kube-api-access-tc5nj\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.629518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.628975    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:51.705180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.705144    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.742442    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.742603    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.742683    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.742875    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5nj\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-kube-api-access-tc5nj\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.742977    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b772da3e-1119-4346-aa14-d376a1f137df-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.743030    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.743139    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.743192    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b772da3e-1119-4346-aa14-d376a1f137df-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.743234    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.743836    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.771742    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b772da3e-1119-4346-aa14-d376a1f137df-istiod-ca-cert\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.772037    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-credential-socket\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.772283    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-data\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.788858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.772927    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-certs\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.919143 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:51.912532    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-envoy\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:51.975592 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:57:51.967771    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:57:52.440004 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.434851    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"]
Nov 02 13:57:52.440004 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.435708    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-istio-token\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:52.440004 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.436165    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b772da3e-1119-4346-aa14-d376a1f137df-istio-podinfo\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:52.440004 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.436219    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5nj\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-kube-api-access-tc5nj\") pod \"e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") " pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:52.461760 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.461669    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:52.462012 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.461986    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:52.462165 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.462147    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:52.497805 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:52.497764    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421a53e0_3b6f_42a5_8256_71e356f87c49.slice/crio-3583706ba8dbcaa5aaf21c71874d0054fb9ff482e5aef0c1e80fa283e524a67b WatchSource:0}: Error finding container 3583706ba8dbcaa5aaf21c71874d0054fb9ff482e5aef0c1e80fa283e524a67b: Status 404 returned error can't find the container with id 3583706ba8dbcaa5aaf21c71874d0054fb9ff482e5aef0c1e80fa283e524a67b
Nov 02 13:57:52.515584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.508366    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"]
Nov 02 13:57:52.547582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.523026    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:52.547582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.523123    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:52.547582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.523175    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:52.598585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.591285    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:52.664775 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.664740    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:52.673181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.672811    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:52.862593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:52.856820    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:53.434498 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.405277    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:53.469727 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.466052    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" event={"ID":"af70e8d4-a1b3-43de-a8e4-f64df1a61613","Type":"ContainerStarted","Data":"47884108c8799d8808593e5b288a49dc911921c426be6dc91532ca3b0b97bf2c"}
Nov 02 13:57:53.469727 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.468800    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" event={"ID":"421a53e0-3b6f-42a5-8256-71e356f87c49","Type":"ContainerStarted","Data":"3583706ba8dbcaa5aaf21c71874d0054fb9ff482e5aef0c1e80fa283e524a67b"}
Nov 02 13:57:53.475500 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.475479    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:53.761272 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:57:53.761143    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe76bd4_674d_44a9_923f_2e473cb6e182.slice/crio-215e4c1f53d31870c5b6842c2be6bac19858809335ecff91764cc6e148cec839 WatchSource:0}: Error finding container 215e4c1f53d31870c5b6842c2be6bac19858809335ecff91764cc6e148cec839: Status 404 returned error can't find the container with id 215e4c1f53d31870c5b6842c2be6bac19858809335ecff91764cc6e148cec839
Nov 02 13:57:53.787600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.787441    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-xks-ca\") pod \"e3905545-6743-4511-9d4c-8fc0bacaafdc\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") "
Nov 02 13:57:53.799932 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.793023    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-openshift-ca\") pod \"e3905545-6743-4511-9d4c-8fc0bacaafdc\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") "
Nov 02 13:57:53.799932 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.794819    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-openshift-ca" pod "e3905545-6743-4511-9d4c-8fc0bacaafdc" (UID: "e3905545-6743-4511-9d4c-8fc0bacaafdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:53.835060 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.833615    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.835060 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.833707    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.835060 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.833754    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.844734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.837167    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.844734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.837247    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.844734 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.837293    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.862766 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.862709    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-xks-ca" pod "e3905545-6743-4511-9d4c-8fc0bacaafdc" (UID: "e3905545-6743-4511-9d4c-8fc0bacaafdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:53.887588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.886957    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"]
Nov 02 13:57:53.887588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.887020    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"]
Nov 02 13:57:53.887588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.887051    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"]
Nov 02 13:57:53.895306 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.895093    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3905545-6743-4511-9d4c-8fc0bacaafdc-kube-api-access-wvmjt\" (UniqueName: \"kubernetes.io/projected/e3905545-6743-4511-9d4c-8fc0bacaafdc-kube-api-access-wvmjt\") pod \"e3905545-6743-4511-9d4c-8fc0bacaafdc\" (UID: \"e3905545-6743-4511-9d4c-8fc0bacaafdc\") "
Nov 02 13:57:53.895661 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.895600    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:53.895661 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.895641    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3905545-6743-4511-9d4c-8fc0bacaafdc-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:53.901446 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.901390    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3905545-6743-4511-9d4c-8fc0bacaafdc-kube-api-access-wvmjt" pod "e3905545-6743-4511-9d4c-8fc0bacaafdc" (UID: "e3905545-6743-4511-9d4c-8fc0bacaafdc"). InnerVolumeSpecName "kube-api-access-wvmjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:53.918664 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.911895    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.918664 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.911993    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:53.918664 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:53.912047    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:57:54.003731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.003696    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmjt\" (UniqueName: \"kubernetes.io/projected/e3905545-6743-4511-9d4c-8fc0bacaafdc-kube-api-access-wvmjt\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:57:54.490584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.490438    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" event={"ID":"0fe76bd4-674d-44a9-923f-2e473cb6e182","Type":"ContainerStarted","Data":"6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1"}
Nov 02 13:57:54.514583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.492252    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" event={"ID":"0fe76bd4-674d-44a9-923f-2e473cb6e182","Type":"ContainerStarted","Data":"215e4c1f53d31870c5b6842c2be6bac19858809335ecff91764cc6e148cec839"}
Nov 02 13:57:54.527225 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.526297    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" event={"ID":"8a5c27fc-880e-498d-bd52-7b1d208f2de4","Type":"ContainerStarted","Data":"586a3302b8cb024112b311c706d47fef1886c33185052d8c9b279d50efb14470"}
Nov 02 13:57:54.527225 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.526366    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" event={"ID":"8a5c27fc-880e-498d-bd52-7b1d208f2de4","Type":"ContainerStarted","Data":"7121f33343caf5ea6bb84833e2acd98b71455c02497dd24724f744909de4e79d"}
Nov 02 13:57:54.543603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.529730    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw" event={"ID":"e3905545-6743-4511-9d4c-8fc0bacaafdc","Type":"ContainerDied","Data":"ec9cc6fbfd376199dd82aeb87b7a33706f93138951029c573c8287e1b29f151a"}
Nov 02 13:57:54.543603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.529780    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec9cc6fbfd376199dd82aeb87b7a33706f93138951029c573c8287e1b29f151a"
Nov 02 13:57:54.543603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.529843    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"
Nov 02 13:57:54.563361 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.563328    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" event={"ID":"421a53e0-3b6f-42a5-8256-71e356f87c49","Type":"ContainerStarted","Data":"d6e249ce398629d2ea1a58c5399e01eddb3895ebd329f9981edbf1b449e89a3d"}
Nov 02 13:57:54.566105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.566079    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" event={"ID":"af70e8d4-a1b3-43de-a8e4-f64df1a61613","Type":"ContainerStarted","Data":"6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b"}
Nov 02 13:57:54.572404 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.572198    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" event={"ID":"b772da3e-1119-4346-aa14-d376a1f137df","Type":"ContainerStarted","Data":"743505720ebb23a21abff1ef377277032c8689e6d3bc80d7aa0090d898b14603"}
Nov 02 13:57:54.572404 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.572242    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" event={"ID":"b772da3e-1119-4346-aa14-d376a1f137df","Type":"ContainerStarted","Data":"d18175e740addadc582c6870c55ac4435dfc87111cfe37d588ebcd46b1a3f40a"}
Nov 02 13:57:54.609019 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.608982    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:57:54.647919 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.647849    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:57:54.900583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.895914    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:57:54.960877 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.960682    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:57:54.965143 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.961178    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:54.965143 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:54.961241    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:57:55.097604 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.097341    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:57:55.148584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.109296    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:57:55.148584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.109372    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:57:55.421615 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.404967    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podStartSLOduration=5.404942463 podStartE2EDuration="5.404942463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:54.873896852 +0000 UTC m=+851.451985720" watchObservedRunningTime="2026-11-02 13:57:55.404942463 +0000 UTC m=+851.983031331"
Nov 02 13:57:55.575037 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.574987    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:55.628607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.616589    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:55.677347 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.677253    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:57:55.677660 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.677616    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:57:55.712473 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.712435    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:57:55.730910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.730867    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:57:55.740332 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.740295    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:55.740685 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.740656    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:57:55.877798 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.877747    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.001643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.895822    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.021978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.903548    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.021978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.926012    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.021978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.926050    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:57:56.021978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:55.977149    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:57:56.050862 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.050728    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.130586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.129684    5572 patch_prober.go:29] interesting pod/thanos-querier-55c454df6-wgrrz container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded" start-of-body=
Nov 02 13:57:56.130586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.129795    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" podUID="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded"
Nov 02 13:57:56.130586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.129891    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:57:56.130586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.130333    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:57:56.130586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.130405    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:57:56.142583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.132179    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:56.142583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.132235    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:57:56.227621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.226791    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.230331 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.230267    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podStartSLOduration=5.230247193 podStartE2EDuration="5.230247193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:54.928613089 +0000 UTC m=+851.506701958" watchObservedRunningTime="2026-11-02 13:57:56.230247193 +0000 UTC m=+852.808336063"
Nov 02 13:57:56.321542 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.321473    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podStartSLOduration=6.321448779 podStartE2EDuration="6.321448779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:54.959251862 +0000 UTC m=+851.537340731" watchObservedRunningTime="2026-11-02 13:57:56.321448779 +0000 UTC m=+852.899537648"
Nov 02 13:57:56.393687 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.393603    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podStartSLOduration=6.393578099 podStartE2EDuration="6.393578099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:54.994177212 +0000 UTC m=+851.572266080" watchObservedRunningTime="2026-11-02 13:57:56.393578099 +0000 UTC m=+852.971666958"
Nov 02 13:57:56.450586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.424091    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podStartSLOduration=6.424071776 podStartE2EDuration="6.424071776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:55.119171051 +0000 UTC m=+851.697259919" watchObservedRunningTime="2026-11-02 13:57:56.424071776 +0000 UTC m=+853.002160648"
Nov 02 13:57:56.565423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.565378    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:57:56.593333 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.593233    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:57:56.594484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.594458    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:57:56.595163 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.595138    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:57:56.599704 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.599674    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.600141 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.600124    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:57:56.621588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.621520    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.622161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.622139    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:57:56.665589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.650927    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:56.665589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.650982    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:57:56.743587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.722500    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:57:56.744461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.744418    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:57:56.902642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.889598    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.915576 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.909672    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:56.954464 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.954428    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:56.954743 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:56.954713    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:57:57.020164 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.009664    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:57.020164 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.019490    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:57:57.108594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.020994    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:57.108594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.107937    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:57:57.120589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.109176    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:57.120589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.109466    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:57:57.120589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.109529    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:57:57.120589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.109597    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:57:57.133124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.130585    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:57.133124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.131856    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:57:57.156302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.144141    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:57:57.156302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.144191    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:57:57.616819 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.604207    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:57.619717 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.619670    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:57:57.786584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.629721    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:57:57.786584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.784030    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:57.788077 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.788051    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:57:57.788260 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.788247    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:57:57.808581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.800456    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:57.808581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.800520    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:57:57.808581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.800640    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:57:57.808581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.800663    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:57:57.808581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.803807    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:57:57.808581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.803855    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:57:57.860155 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.860111    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:57.938412 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:57.938318    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.036606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.019109    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.036606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.027788    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.088970 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.088935    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:57:58.089249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.089220    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:57:58.091684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.091660    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:58.091859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.091838    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:57:58.137628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.117107    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.137628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.119012    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.138552 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.138518    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.139053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.139027    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.165350 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.165311    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.607586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.604424    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.607823 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.607798    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:57:58.608820 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.608792    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:57:58.656593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.645285    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:58.656875 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.656841    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:57:58.711602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.711159    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:57:58.711602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.711219    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:57:58.781909 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.781869    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.804351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.802327    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.804351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.802599    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.804351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.802862    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:57:58.804351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.803065    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:57:58.846720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.809006    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:57:58.899707 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.899611    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"]
Nov 02 13:57:58.903950 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.903919    5572 cpu_manager.go:414] "containerMap: removing container" podUID="e3905545-6743-4511-9d4c-8fc0bacaafdc" containerName="revoke-keys"
Nov 02 13:57:58.904601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.904551    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3905545-6743-4511-9d4c-8fc0bacaafdc" containerName="revoke-keys"
Nov 02 13:57:58.905071 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.905052    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3905545-6743-4511-9d4c-8fc0bacaafdc" containerName="revoke-keys"
Nov 02 13:57:58.922632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.920541    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:58.938672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.938644    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1eb567\"/\"distinct-2-w6-1eb567-kserve-self-signed-certs\""
Nov 02 13:57:58.958482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.958214    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:58.958482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:58.958281    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.006727    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tls-certs\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.006814    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-dshm\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.006851    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-home\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.006905    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ll4n\" (UniqueName: \"kubernetes.io/projected/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-kube-api-access-4ll4n\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.006991    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tmp-dir\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.007179    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-model-cache\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.008780    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:59.010598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.009230    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:57:59.031675 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.029070    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:59.031675 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.029419    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:57:59.116588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.108104    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:57:59.116588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.108220    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:57:59.173651 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.161376    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:57:59.173651 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.162057    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.247336    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tls-certs\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.247420    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-dshm\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.247459    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-home\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.247514    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ll4n\" (UniqueName: \"kubernetes.io/projected/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-kube-api-access-4ll4n\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.247639    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tmp-dir\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.247814    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-model-cache\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.248462    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-model-cache\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.274201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.264707    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:57:59.424644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.405380    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tmp-dir\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.451841 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.418068    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-home\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.524605 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.524520    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ll4n\" (UniqueName: \"kubernetes.io/projected/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-kube-api-access-4ll4n\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.570827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.570782    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tls-certs\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.601260 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.601226    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:57:59.601604 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.601535    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:57:59.666362 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.666326    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:59.666669 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.666638    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:57:59.701683 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.689991    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-dshm\") pod \"distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") " pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:57:59.884209 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.884171    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:57:59.884510 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.884477    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:57:59.907415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.907369    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:57:59.931272 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.931234    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:57:59.945911 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.945850    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:00.031833 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:57:59.954316    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:58:00.054665 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.005331    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:58:00.054999 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.054948    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:58:00.055160 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.030469    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:00.248591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.028919    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:00.276643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.030517    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:58:00.278085 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.278051    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:58:00.278282 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.278265    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"]
Nov 02 13:58:00.290707 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.290631    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:58:00.291927 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.291898    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:58:00.345595 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.305697    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:58:00.485594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.351370    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:58:00.528609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.352689    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:58:00.687201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.687158    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:58:00.687525 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.687496    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:58:00.688355 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.688312    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:00.688530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.688508    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:58:00.690710 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.690684    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded"
Nov 02 13:58:00.824834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.824800    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:58:00.826749 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.826719    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:58:00.826926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.825490    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:58:00.827079 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.827060    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:58:00.936600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.935782    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:58:00.994643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.979753    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:58:00.994643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.979815    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:58:00.994643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:00.980188    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:58:01.027392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.023221    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:58:01.052780 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.032714    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:01.056696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.056659    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:01.079133 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.078765    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:01.253647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.247684    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:01.290587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.277142    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:58:01.407334 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.338935    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:01.485689 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.399643    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:58:01.569825 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.569777    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:01.684245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.668185    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:01.684245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.668246    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:58:01.684245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.675465    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:58:01.684245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.675523    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:58:01.762960 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.762921    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:58:01.763237 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.763210    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:58:01.956717 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:01.956666    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.122866 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.122819    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.156682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.156619    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:02.156850 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.156728    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.168930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.158852    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:58:02.168930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.158902    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:58:02.189234 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.179524    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:58:02.189234 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.179616    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:58:02.202609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.202244    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.279183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.242582    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:58:02.279183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.252601    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:58:02.279183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.253324    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.279183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.262228    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:58:02.281797 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.281756    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.292424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.186951    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:58:02.514769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.514657    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.552586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.517059    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:02.647730 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.647680    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:58:02.686593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.686538    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:58:02.704692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.704652    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:58:02.704946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.688476    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:02.705078 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.705055    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:58:02.731586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.706360    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 02 13:58:02.731949 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.707313    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:58:02.732094 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.732066    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:58:02.801655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.800711    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:58:02.801655 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.800763    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"]
Nov 02 13:58:02.979586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.977682    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:58:02.979586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.977750    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 13:58:02.979586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:02.978741    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.134588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.132170    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:58:03.134588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.132228    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:58:03.134588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.132512    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.134588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.132867    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:58:03.217914 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.217867    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.262609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.262366    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:58:03.274603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.267386    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:58:03.274603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.267772    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:58:03.274603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.268033    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.274603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.268310    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:58:03.322814 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.301702    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.322814 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.302399    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.451674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.383253    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.527601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.486512    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" event={"ID":"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e","Type":"ContainerStarted","Data":"fbf49cf0e01dd77573eb63bab30cfd08fe2a766459274449a1ee9a9907e514b5"}
Nov 02 13:58:03.579692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.527357    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.580221 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.553718    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.601593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.596906    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:03.601593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.596965    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:58:03.651659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.639892    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:58:03.651659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.639947    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:58:03.651659 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.650084    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:03.720321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.720230    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:58:03.722554 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.722516    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:58:03.980700 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:03.978907    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:04.095689 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.089658    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"]
Nov 02 13:58:04.119584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.115689    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 13:58:04.119584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.115749    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 13:58:04.165954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.165914    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:04.429005 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.428798    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:58:04.458588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.446768    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:58:04.492906 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.477024    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:04.494657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.478670    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:04.495160 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.478716    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:58:04.495613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.492242    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:04.522717 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.522667    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:58:04.614419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.614311    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:04.623989 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.623959    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:04.624231 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.624209    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:58:04.653512 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.651532    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:04.669253 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.614696    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:58:04.675362 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.675319    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:58:04.675625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.675602    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:58:04.708584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.695811    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:58:04.708584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.695899    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:58:04.708584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.696094    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:58:04.708584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.696113    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" event={"ID":"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e","Type":"ContainerStarted","Data":"cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c"}
Nov 02 13:58:04.708584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.696141    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"]
Nov 02 13:58:04.708584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.696438    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:04.717981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.717951    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 13:58:04.718249 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.718205    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 13:58:04.718866 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.718843    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:58:04.719140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.719123    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:58:04.727453 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.727428    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:58:04.730802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.730781    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:04.731630 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.731612    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:58:04.762594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.738998    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx98v\" (UniqueName: \"kubernetes.io/projected/2e1078fd-5ea3-4888-ac87-787f791cc6dd-kube-api-access-xx98v\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:04.765373 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.765336    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:04.804064 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.804034    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 13:58:04.816188 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.816126    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podStartSLOduration=6.816106503 podStartE2EDuration="6.816106503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:04.727635327 +0000 UTC m=+861.305724195" watchObservedRunningTime="2026-11-02 13:58:04.816106503 +0000 UTC m=+861.394195371"
Nov 02 13:58:04.855112 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.854969    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:04.899901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.899818    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:58:04.981535 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.979986    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:58:04.989644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.988828    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:58:04.992578 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:04.992488    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:05.047645 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.047600    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:05.056619 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.048211    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx98v\" (UniqueName: \"kubernetes.io/projected/2e1078fd-5ea3-4888-ac87-787f791cc6dd-kube-api-access-xx98v\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:05.063767 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.063727    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:05.073624 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.072904    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:05.222717 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.222609    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:05.288238 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.287966    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:58:05.306139 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.306111    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:58:05.330413 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.330384    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:58:05.341540 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.312267    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx98v\" (UniqueName: \"kubernetes.io/projected/2e1078fd-5ea3-4888-ac87-787f791cc6dd-kube-api-access-xx98v\") pod \"maas-api-revoke-sub-97c4f20f-4n8tl\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:05.354073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.354017    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:58:05.405908 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.405876    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:58:05.472623 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.471702    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:05.576232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.576211    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:58:05.597287 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.597247    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:05.597446 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.597296    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 13:58:05.674516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.660578    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:58:05.674516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.660634    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 02 13:58:05.820507 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.820476    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:58:05.822673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.822210    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:58:05.848868 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.848842    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 13:58:05.952699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.952097    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:58:05.953706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.953047    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:58:05.959603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:05.955888    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 13:58:06.010300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.010266    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"]
Nov 02 13:58:06.639055 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.639019    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:58:06.643482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.643454    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:58:06.710929 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.710891    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl" event={"ID":"2e1078fd-5ea3-4888-ac87-787f791cc6dd","Type":"ContainerStarted","Data":"5bacd5938f87e9c43131f58f0b2ee99b665699019b66e6a7ccc2bd8a39e6c231"}
Nov 02 13:58:06.711098 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.710949    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl" event={"ID":"2e1078fd-5ea3-4888-ac87-787f791cc6dd","Type":"ContainerStarted","Data":"d1b09fa0c2e9bd22b118e8bd078d9e207c160e2f8878f2642922cc6362c39b05"}
Nov 02 13:58:06.713035 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.713007    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 13:58:06.735755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.735702    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl" podStartSLOduration=3.735687468 podStartE2EDuration="3.735687468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:06.729737924 +0000 UTC m=+863.307826793" watchObservedRunningTime="2026-11-02 13:58:06.735687468 +0000 UTC m=+863.313776335"
Nov 02 13:58:06.754752 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.754722    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:58:06.755754 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.755729    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:58:06.757120 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:06.757093    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 13:58:07.720146 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:07.720108    5572 generic.go:356] "Generic (PLEG): container finished" podID="2e1078fd-5ea3-4888-ac87-787f791cc6dd" containerID="5bacd5938f87e9c43131f58f0b2ee99b665699019b66e6a7ccc2bd8a39e6c231" exitCode=0
Nov 02 13:58:07.720621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:07.720203    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl" event={"ID":"2e1078fd-5ea3-4888-ac87-787f791cc6dd","Type":"ContainerDied","Data":"5bacd5938f87e9c43131f58f0b2ee99b665699019b66e6a7ccc2bd8a39e6c231"}
Nov 02 13:58:08.894666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.894642    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:08.932546 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.932512    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-xks-ca\") pod \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") "
Nov 02 13:58:08.932783 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.932693    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-openshift-ca\") pod \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") "
Nov 02 13:58:08.933006 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.932980    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1078fd-5ea3-4888-ac87-787f791cc6dd-kube-api-access-xx98v\" (UniqueName: \"kubernetes.io/projected/2e1078fd-5ea3-4888-ac87-787f791cc6dd-kube-api-access-xx98v\") pod \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\" (UID: \"2e1078fd-5ea3-4888-ac87-787f791cc6dd\") "
Nov 02 13:58:08.933126 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.933097    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-openshift-ca" pod "2e1078fd-5ea3-4888-ac87-787f791cc6dd" (UID: "2e1078fd-5ea3-4888-ac87-787f791cc6dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:08.933833 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.933811    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:08.935193 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.935170    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-xks-ca" pod "2e1078fd-5ea3-4888-ac87-787f791cc6dd" (UID: "2e1078fd-5ea3-4888-ac87-787f791cc6dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:08.935458 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:08.935436    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1078fd-5ea3-4888-ac87-787f791cc6dd-kube-api-access-xx98v" pod "2e1078fd-5ea3-4888-ac87-787f791cc6dd" (UID: "2e1078fd-5ea3-4888-ac87-787f791cc6dd"). InnerVolumeSpecName "kube-api-access-xx98v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:09.036034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:09.035962    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx98v\" (UniqueName: \"kubernetes.io/projected/2e1078fd-5ea3-4888-ac87-787f791cc6dd-kube-api-access-xx98v\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:09.036034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:09.035990    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e1078fd-5ea3-4888-ac87-787f791cc6dd-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:09.738358 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:09.738330    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"
Nov 02 13:58:09.738527 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:09.738356    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl" event={"ID":"2e1078fd-5ea3-4888-ac87-787f791cc6dd","Type":"ContainerDied","Data":"d1b09fa0c2e9bd22b118e8bd078d9e207c160e2f8878f2642922cc6362c39b05"}
Nov 02 13:58:09.738527 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:09.738402    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b09fa0c2e9bd22b118e8bd078d9e207c160e2f8878f2642922cc6362c39b05"
Nov 02 13:58:10.354698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:10.354667    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:58:10.367093 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:10.367068    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:58:10.765873 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:10.765790    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:58:13.267040 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.267004    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"]
Nov 02 13:58:13.268109 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.268089    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2e1078fd-5ea3-4888-ac87-787f791cc6dd" containerName="revoke-keys"
Nov 02 13:58:13.268158 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.268113    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1078fd-5ea3-4888-ac87-787f791cc6dd" containerName="revoke-keys"
Nov 02 13:58:13.268308 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.268292    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1078fd-5ea3-4888-ac87-787f791cc6dd" containerName="revoke-keys"
Nov 02 13:58:13.272974 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.272952    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.278210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.278189    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9e0c2f\"/\"openshift-service-ca.crt\""
Nov 02 13:58:13.279210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.279185    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9e0c2f\"/\"default-dockercfg-kmfmt\""
Nov 02 13:58:13.279382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.279359    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9e0c2f\"/\"distinct-2-w4-9e0c2f-kserve-self-signed-certs\""
Nov 02 13:58:13.281107 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.281077    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9e0c2f\"/\"kube-root-ca.crt\""
Nov 02 13:58:13.282036 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.281873    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"]
Nov 02 13:58:13.431946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.431916    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-home\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.431946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.431945    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-dshm\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.432159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.432143    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-model-cache\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.432259 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.432244    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dr62\" (UniqueName: \"kubernetes.io/projected/c12e3288-3bdd-4ca6-97cf-5d5af6062346-kube-api-access-5dr62\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.432311 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.432299    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tls-certs\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.432357 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.432347    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tmp-dir\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.533535 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.533450    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-model-cache\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.533729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.533585    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dr62\" (UniqueName: \"kubernetes.io/projected/c12e3288-3bdd-4ca6-97cf-5d5af6062346-kube-api-access-5dr62\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.533729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.533654    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tls-certs\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.533729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.533693    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tmp-dir\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.533921 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.533783    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-home\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.533921 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.533804    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-dshm\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.533921 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.533906    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-model-cache\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.534102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.534078    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-home\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.534237 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.534081    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tmp-dir\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.536212 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.536189    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-dshm\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.536812 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.536790    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tls-certs\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.565926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.565879    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dr62\" (UniqueName: \"kubernetes.io/projected/c12e3288-3bdd-4ca6-97cf-5d5af6062346-kube-api-access-5dr62\") pod \"distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") " pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:13.594997 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:13.594945    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:14.054943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:14.054919    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"]
Nov 02 13:58:14.057452 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:14.057424    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc12e3288_3bdd_4ca6_97cf_5d5af6062346.slice/crio-543d8429a6388e756c5ceaf39fbe346a2a3f4a0b7a6fcb72a4a8a85f2db628d0 WatchSource:0}: Error finding container 543d8429a6388e756c5ceaf39fbe346a2a3f4a0b7a6fcb72a4a8a85f2db628d0: Status 404 returned error can't find the container with id 543d8429a6388e756c5ceaf39fbe346a2a3f4a0b7a6fcb72a4a8a85f2db628d0
Nov 02 13:58:14.801079 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:14.801033    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" event={"ID":"c12e3288-3bdd-4ca6-97cf-5d5af6062346","Type":"ContainerStarted","Data":"e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe"}
Nov 02 13:58:14.801521 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:14.801089    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" event={"ID":"c12e3288-3bdd-4ca6-97cf-5d5af6062346","Type":"ContainerStarted","Data":"543d8429a6388e756c5ceaf39fbe346a2a3f4a0b7a6fcb72a4a8a85f2db628d0"}
Nov 02 13:58:14.838008 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:14.835914    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podStartSLOduration=1.835891629 podStartE2EDuration="1.835891629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:14.832959014 +0000 UTC m=+871.411047885" watchObservedRunningTime="2026-11-02 13:58:14.835891629 +0000 UTC m=+871.413980497"
Nov 02 13:58:15.596104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:15.596074    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:16.122310 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.094785    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"]
Nov 02 13:58:16.153982 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.132904    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.174581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.171141    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:16.174581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.171431    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:58:16.174581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.172646    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810bd192-0081-4082-9408-2eda4b79727d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.174581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.173084    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810bd192-0081-4082-9408-2eda4b79727d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.174581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.173179    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdxg\" (UniqueName: \"kubernetes.io/projected/810bd192-0081-4082-9408-2eda4b79727d-kube-api-access-crdxg\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.225993 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.225910    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"]
Nov 02 13:58:16.308622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.288381    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810bd192-0081-4082-9408-2eda4b79727d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.308622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.297713    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdxg\" (UniqueName: \"kubernetes.io/projected/810bd192-0081-4082-9408-2eda4b79727d-kube-api-access-crdxg\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.308622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.297923    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810bd192-0081-4082-9408-2eda4b79727d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.308622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.293759    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810bd192-0081-4082-9408-2eda4b79727d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.328585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.327813    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810bd192-0081-4082-9408-2eda4b79727d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.340637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.331643    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdxg\" (UniqueName: \"kubernetes.io/projected/810bd192-0081-4082-9408-2eda4b79727d-kube-api-access-crdxg\") pod \"maas-api-revoke-sub-18d90261-r4q72\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:16.526415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.526324    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:17.084587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:16.954671    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"]
Nov 02 13:58:17.759205 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:17.759161    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:17.795608 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:17.795216    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 13:58:17.805591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:17.802638    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:17.821602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:17.820636    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:17.821602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:17.820636    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:18.049654 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.048809    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:18.078617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.050059    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:18.085777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.085637    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:18.152588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.149459    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:18.159726 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.155430    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72" event={"ID":"810bd192-0081-4082-9408-2eda4b79727d","Type":"ContainerStarted","Data":"bb810230887e8ddfd86a55436f2eaf0ad7ae9be91de6f82cf42685fb3febf892"}
Nov 02 13:58:18.159726 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.155494    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72" event={"ID":"810bd192-0081-4082-9408-2eda4b79727d","Type":"ContainerStarted","Data":"595c4634b12ac9bde128081ab7eca53fbb81087289e48d38c8ce6838104bd4a7"}
Nov 02 13:58:18.228628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.220664    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:18.230667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.229465    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72" podStartSLOduration=3.22944499 podStartE2EDuration="3.22944499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:18.202864143 +0000 UTC m=+874.780953011" watchObservedRunningTime="2026-11-02 13:58:18.22944499 +0000 UTC m=+874.807533857"
Nov 02 13:58:18.230667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:18.230229    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:19.178748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:19.178703    5572 generic.go:356] "Generic (PLEG): container finished" podID="810bd192-0081-4082-9408-2eda4b79727d" containerID="bb810230887e8ddfd86a55436f2eaf0ad7ae9be91de6f82cf42685fb3febf892" exitCode=0
Nov 02 13:58:19.179195 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:19.178797    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72" event={"ID":"810bd192-0081-4082-9408-2eda4b79727d","Type":"ContainerDied","Data":"bb810230887e8ddfd86a55436f2eaf0ad7ae9be91de6f82cf42685fb3febf892"}
Nov 02 13:58:19.987917 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:19.987881    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"]
Nov 02 13:58:19.992895 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:19.992872    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.006851 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.003356    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"]
Nov 02 13:58:20.174260 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.174222    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zc5b\" (UniqueName: \"kubernetes.io/projected/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-kube-api-access-6zc5b\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.174260 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.174264    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.174505 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.174375    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.275972 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.275904    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zc5b\" (UniqueName: \"kubernetes.io/projected/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-kube-api-access-6zc5b\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.276405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.275992    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.276405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.276050    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.276701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.276673    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.289522 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.289493    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zc5b\" (UniqueName: \"kubernetes.io/projected/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-kube-api-access-6zc5b\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.289930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.289907    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b0e8b9-wzwkk\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.312438 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.312415    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:20.361863 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.361842    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:20.377531 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.377491    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810bd192-0081-4082-9408-2eda4b79727d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810bd192-0081-4082-9408-2eda4b79727d-maas-api-openshift-ca\") pod \"810bd192-0081-4082-9408-2eda4b79727d\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") "
Nov 02 13:58:20.377707 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.377650    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/810bd192-0081-4082-9408-2eda4b79727d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810bd192-0081-4082-9408-2eda4b79727d-maas-api-xks-ca\") pod \"810bd192-0081-4082-9408-2eda4b79727d\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") "
Nov 02 13:58:20.377855 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.377836    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810bd192-0081-4082-9408-2eda4b79727d-kube-api-access-crdxg\" (UniqueName: \"kubernetes.io/projected/810bd192-0081-4082-9408-2eda4b79727d-kube-api-access-crdxg\") pod \"810bd192-0081-4082-9408-2eda4b79727d\" (UID: \"810bd192-0081-4082-9408-2eda4b79727d\") "
Nov 02 13:58:20.377926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.377855    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810bd192-0081-4082-9408-2eda4b79727d-maas-api-openshift-ca" pod "810bd192-0081-4082-9408-2eda4b79727d" (UID: "810bd192-0081-4082-9408-2eda4b79727d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:20.378540 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.378516    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810bd192-0081-4082-9408-2eda4b79727d-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:20.380096 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.380066    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/810bd192-0081-4082-9408-2eda4b79727d-maas-api-xks-ca" pod "810bd192-0081-4082-9408-2eda4b79727d" (UID: "810bd192-0081-4082-9408-2eda4b79727d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:20.380598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.380536    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810bd192-0081-4082-9408-2eda4b79727d-kube-api-access-crdxg" pod "810bd192-0081-4082-9408-2eda4b79727d" (UID: "810bd192-0081-4082-9408-2eda4b79727d"). InnerVolumeSpecName "kube-api-access-crdxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:20.479682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.479647    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810bd192-0081-4082-9408-2eda4b79727d-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:20.479682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.479682    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crdxg\" (UniqueName: \"kubernetes.io/projected/810bd192-0081-4082-9408-2eda4b79727d-kube-api-access-crdxg\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:20.600917 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:20.600885    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe3461a_0791_4ed6_b2f7_e07b5c01299d.slice/crio-719b57f64be794952a4e365ab3d3cf1a44f6088f1fae9af48d49f45d527e03df WatchSource:0}: Error finding container 719b57f64be794952a4e365ab3d3cf1a44f6088f1fae9af48d49f45d527e03df: Status 404 returned error can't find the container with id 719b57f64be794952a4e365ab3d3cf1a44f6088f1fae9af48d49f45d527e03df
Nov 02 13:58:20.601706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:20.601684    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"]
Nov 02 13:58:21.080641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.080608    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"]
Nov 02 13:58:21.081396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.081375    5572 cpu_manager.go:414] "containerMap: removing container" podUID="810bd192-0081-4082-9408-2eda4b79727d" containerName="revoke-keys"
Nov 02 13:58:21.081396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.081396    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810bd192-0081-4082-9408-2eda4b79727d" containerName="revoke-keys"
Nov 02 13:58:21.081662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.081639    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="810bd192-0081-4082-9408-2eda4b79727d" containerName="revoke-keys"
Nov 02 13:58:21.085885 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.085859    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.108218 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.108190    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"]
Nov 02 13:58:21.191233 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191201    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191257    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191280    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191390    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191512 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191421    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191512 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191485    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r95p\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-kube-api-access-5r95p\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191619 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191602    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82d4e2b-80d9-4195-827e-035f0b5f026a-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191668 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191656    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.191762 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.191750    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.195577 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.195534    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72" event={"ID":"810bd192-0081-4082-9408-2eda4b79727d","Type":"ContainerDied","Data":"595c4634b12ac9bde128081ab7eca53fbb81087289e48d38c8ce6838104bd4a7"}
Nov 02 13:58:21.195731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.195603    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"
Nov 02 13:58:21.195731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.195605    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="595c4634b12ac9bde128081ab7eca53fbb81087289e48d38c8ce6838104bd4a7"
Nov 02 13:58:21.197482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.197458    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk" event={"ID":"8fe3461a-0791-4ed6-b2f7-e07b5c01299d","Type":"ContainerStarted","Data":"ef5ae095a02262e498ffdd19384c9ff5e07344916f00ab40c15fc9bc3386bebd"}
Nov 02 13:58:21.197628 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.197500    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk" event={"ID":"8fe3461a-0791-4ed6-b2f7-e07b5c01299d","Type":"ContainerStarted","Data":"719b57f64be794952a4e365ab3d3cf1a44f6088f1fae9af48d49f45d527e03df"}
Nov 02 13:58:21.225439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.225397    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk" podStartSLOduration=2.225385963 podStartE2EDuration="2.225385963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:21.221092987 +0000 UTC m=+877.799181855" watchObservedRunningTime="2026-11-02 13:58:21.225385963 +0000 UTC m=+877.803474830"
Nov 02 13:58:21.293458 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293380    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.293458 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293449    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.293946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293479    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.293946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293532    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.293946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293582    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.293946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293668    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r95p\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-kube-api-access-5r95p\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.293946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293854    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.293946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293887    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-credential-socket\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.294257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.293971    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82d4e2b-80d9-4195-827e-035f0b5f026a-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.294257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.294044    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.294257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.294119    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.294506 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.294481    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-data\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.294626 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.294611    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-certs\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.295045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.295019    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82d4e2b-80d9-4195-827e-035f0b5f026a-istiod-ca-cert\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.296102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.296079    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-envoy\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.296317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.296299    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-podinfo\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.305985 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.305964    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-token\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.305985 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.305979    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r95p\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-kube-api-access-5r95p\") pod \"e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") " pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.404943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.404918    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:21.435960 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:58:21.435927    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:58:21.735534 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:21.735504    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82d4e2b_80d9_4195_827e_035f0b5f026a.slice/crio-d7737b265b3de56e50bf842f46c94a4848dad5d42b63b248b4caa0a442efc395 WatchSource:0}: Error finding container d7737b265b3de56e50bf842f46c94a4848dad5d42b63b248b4caa0a442efc395: Status 404 returned error can't find the container with id d7737b265b3de56e50bf842f46c94a4848dad5d42b63b248b4caa0a442efc395
Nov 02 13:58:21.737352 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.737328    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"]
Nov 02 13:58:21.739043 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.739009    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:21.739133 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.739090    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:21.739178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:21.739133    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:22.207824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.207789    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" event={"ID":"f82d4e2b-80d9-4195-827e-035f0b5f026a","Type":"ContainerStarted","Data":"cb594dfef9f4b3f42467f53a68b97dea043d455554d854a72a83ce805c05801d"}
Nov 02 13:58:22.208023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.207853    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" event={"ID":"f82d4e2b-80d9-4195-827e-035f0b5f026a","Type":"ContainerStarted","Data":"d7737b265b3de56e50bf842f46c94a4848dad5d42b63b248b4caa0a442efc395"}
Nov 02 13:58:22.209697 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.209671    5572 generic.go:356] "Generic (PLEG): container finished" podID="8fe3461a-0791-4ed6-b2f7-e07b5c01299d" containerID="ef5ae095a02262e498ffdd19384c9ff5e07344916f00ab40c15fc9bc3386bebd" exitCode=0
Nov 02 13:58:22.209840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.209730    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk" event={"ID":"8fe3461a-0791-4ed6-b2f7-e07b5c01299d","Type":"ContainerDied","Data":"ef5ae095a02262e498ffdd19384c9ff5e07344916f00ab40c15fc9bc3386bebd"}
Nov 02 13:58:22.236219 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.236114    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podStartSLOduration=1.236100246 podStartE2EDuration="1.236100246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:22.233360257 +0000 UTC m=+878.811449123" watchObservedRunningTime="2026-11-02 13:58:22.236100246 +0000 UTC m=+878.814189115"
Nov 02 13:58:22.405285 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.405257    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:22.407159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.407130    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 13:58:22.407300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:22.407181    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 13:58:23.406070 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.406023    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 13:58:23.406547 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.406095    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 13:58:23.432981 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.431806    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:23.443227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.443188    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-kube-api-access-6zc5b\" (UniqueName: \"kubernetes.io/projected/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-kube-api-access-6zc5b\") pod \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") "
Nov 02 13:58:23.443394 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.443267    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-openshift-ca\") pod \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") "
Nov 02 13:58:23.443394 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.443320    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-xks-ca\") pod \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\" (UID: \"8fe3461a-0791-4ed6-b2f7-e07b5c01299d\") "
Nov 02 13:58:23.444025 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.443996    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-openshift-ca" pod "8fe3461a-0791-4ed6-b2f7-e07b5c01299d" (UID: "8fe3461a-0791-4ed6-b2f7-e07b5c01299d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:23.444423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.444400    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:23.477640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.476318    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-kube-api-access-6zc5b" pod "8fe3461a-0791-4ed6-b2f7-e07b5c01299d" (UID: "8fe3461a-0791-4ed6-b2f7-e07b5c01299d"). InnerVolumeSpecName "kube-api-access-6zc5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:23.534802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.526178    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-xks-ca" pod "8fe3461a-0791-4ed6-b2f7-e07b5c01299d" (UID: "8fe3461a-0791-4ed6-b2f7-e07b5c01299d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:23.565590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.555094    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:23.565590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.555131    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zc5b\" (UniqueName: \"kubernetes.io/projected/8fe3461a-0791-4ed6-b2f7-e07b5c01299d-kube-api-access-6zc5b\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:23.625376 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.625342    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:23.632124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:23.632042    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:24.281601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:24.278779    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"
Nov 02 13:58:24.281601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:24.278921    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk" event={"ID":"8fe3461a-0791-4ed6-b2f7-e07b5c01299d","Type":"ContainerDied","Data":"719b57f64be794952a4e365ab3d3cf1a44f6088f1fae9af48d49f45d527e03df"}
Nov 02 13:58:24.281601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:24.278964    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="719b57f64be794952a4e365ab3d3cf1a44f6088f1fae9af48d49f45d527e03df"
Nov 02 13:58:24.362609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:24.361865    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:24.405953 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:24.405914    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 13:58:24.406155 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:24.405979    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 13:58:25.406048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:25.406007    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 13:58:25.406248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:25.406070    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 13:58:26.406058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:26.406021    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 13:58:26.406348 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:26.406314    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 13:58:27.406068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:27.406031    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 13:58:27.406250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:27.406098    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 13:58:28.409340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:28.409210    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 13:58:28.410045 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:28.410003    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 13:58:29.471499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:29.471459    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:29.472375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:29.472354    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:29.473533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:29.473511    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 13:58:32.386356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.386313    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"]
Nov 02 13:58:32.387072 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.387057    5572 cpu_manager.go:414] "containerMap: removing container" podUID="8fe3461a-0791-4ed6-b2f7-e07b5c01299d" containerName="revoke-keys"
Nov 02 13:58:32.387132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.387074    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe3461a-0791-4ed6-b2f7-e07b5c01299d" containerName="revoke-keys"
Nov 02 13:58:32.387199 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.387191    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe3461a-0791-4ed6-b2f7-e07b5c01299d" containerName="revoke-keys"
Nov 02 13:58:32.390907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.390889    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.396544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.396520    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:58:32.396715 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.396612    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:32.400809 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.400786    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"]
Nov 02 13:58:32.449194 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.449165    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.449379 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.449326    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4w9\" (UniqueName: \"kubernetes.io/projected/a47c2db8-393e-450f-8dad-28e3cddb860d-kube-api-access-jq4w9\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.449515 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.449498    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.551140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.551104    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.551316 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.551210    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4w9\" (UniqueName: \"kubernetes.io/projected/a47c2db8-393e-450f-8dad-28e3cddb860d-kube-api-access-jq4w9\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.551423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.551329    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.551781 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.551756    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.553897 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.553868    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.565068 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.565040    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4w9\" (UniqueName: \"kubernetes.io/projected/a47c2db8-393e-450f-8dad-28e3cddb860d-kube-api-access-jq4w9\") pod \"maas-api-revoke-sub-397ec389-jwnw5\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:32.717683 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:32.715828    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:33.066412 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:33.066385    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47c2db8_393e_450f_8dad_28e3cddb860d.slice/crio-134a49f6b80797ea3ebebccfb0a8c4c65e4de03fa26e4803a1623ed731a713cd WatchSource:0}: Error finding container 134a49f6b80797ea3ebebccfb0a8c4c65e4de03fa26e4803a1623ed731a713cd: Status 404 returned error can't find the container with id 134a49f6b80797ea3ebebccfb0a8c4c65e4de03fa26e4803a1623ed731a713cd
Nov 02 13:58:33.066896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:33.066873    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"]
Nov 02 13:58:33.410313 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:33.410287    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5" event={"ID":"a47c2db8-393e-450f-8dad-28e3cddb860d","Type":"ContainerStarted","Data":"0ebada1599572d3d64daa8e77c8426c69ec3405b578aa890ac070629feaff073"}
Nov 02 13:58:33.410647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:33.410337    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5" event={"ID":"a47c2db8-393e-450f-8dad-28e3cddb860d","Type":"ContainerStarted","Data":"134a49f6b80797ea3ebebccfb0a8c4c65e4de03fa26e4803a1623ed731a713cd"}
Nov 02 13:58:33.435617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:33.435554    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5" podStartSLOduration=1.435540802 podStartE2EDuration="1.435540802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:33.431020542 +0000 UTC m=+890.009109411" watchObservedRunningTime="2026-11-02 13:58:33.435540802 +0000 UTC m=+890.013629712"
Nov 02 13:58:34.429100 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:34.429062    5572 generic.go:356] "Generic (PLEG): container finished" podID="a47c2db8-393e-450f-8dad-28e3cddb860d" containerID="0ebada1599572d3d64daa8e77c8426c69ec3405b578aa890ac070629feaff073" exitCode=0
Nov 02 13:58:34.429712 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:34.429134    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5" event={"ID":"a47c2db8-393e-450f-8dad-28e3cddb860d","Type":"ContainerDied","Data":"0ebada1599572d3d64daa8e77c8426c69ec3405b578aa890ac070629feaff073"}
Nov 02 13:58:35.369594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.349037    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"]
Nov 02 13:58:35.385408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.378761    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.402518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.402477    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jms4q\" (UniqueName: \"kubernetes.io/projected/ccd49051-1365-4f49-83bd-26765a94eaac-kube-api-access-jms4q\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.402758 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.402550    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.402758 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.402656    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.403209 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.403179    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"]
Nov 02 13:58:35.503657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.503624    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jms4q\" (UniqueName: \"kubernetes.io/projected/ccd49051-1365-4f49-83bd-26765a94eaac-kube-api-access-jms4q\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.504103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.503681    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.504103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.503768    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.504480 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.504434    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.506634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.506613    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.516957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.516928    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jms4q\" (UniqueName: \"kubernetes.io/projected/ccd49051-1365-4f49-83bd-26765a94eaac-kube-api-access-jms4q\") pod \"maas-api-revoke-sub-cb51c58e-9pvsf\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.632147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.632093    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:35.706647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.706611    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47c2db8-393e-450f-8dad-28e3cddb860d-kube-api-access-jq4w9\" (UniqueName: \"kubernetes.io/projected/a47c2db8-393e-450f-8dad-28e3cddb860d-kube-api-access-jq4w9\") pod \"a47c2db8-393e-450f-8dad-28e3cddb860d\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") "
Nov 02 13:58:35.706835 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.706774    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-xks-ca\") pod \"a47c2db8-393e-450f-8dad-28e3cddb860d\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") "
Nov 02 13:58:35.707013 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.706995    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-openshift-ca\") pod \"a47c2db8-393e-450f-8dad-28e3cddb860d\" (UID: \"a47c2db8-393e-450f-8dad-28e3cddb860d\") "
Nov 02 13:58:35.707352 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.707327    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-openshift-ca" pod "a47c2db8-393e-450f-8dad-28e3cddb860d" (UID: "a47c2db8-393e-450f-8dad-28e3cddb860d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:35.707743 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.707722    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:35.709507 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.709486    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-xks-ca" pod "a47c2db8-393e-450f-8dad-28e3cddb860d" (UID: "a47c2db8-393e-450f-8dad-28e3cddb860d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:35.709507 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.709499    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47c2db8-393e-450f-8dad-28e3cddb860d-kube-api-access-jq4w9" pod "a47c2db8-393e-450f-8dad-28e3cddb860d" (UID: "a47c2db8-393e-450f-8dad-28e3cddb860d"). InnerVolumeSpecName "kube-api-access-jq4w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:35.756103 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.756082    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:35.810954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.810458    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4w9\" (UniqueName: \"kubernetes.io/projected/a47c2db8-393e-450f-8dad-28e3cddb860d-kube-api-access-jq4w9\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:35.810954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.810480    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47c2db8-393e-450f-8dad-28e3cddb860d-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:35.994472 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.994400    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"]
Nov 02 13:58:35.995443 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.995417    5572 cpu_manager.go:414] "containerMap: removing container" podUID="a47c2db8-393e-450f-8dad-28e3cddb860d" containerName="revoke-keys"
Nov 02 13:58:35.995443 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.995443    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47c2db8-393e-450f-8dad-28e3cddb860d" containerName="revoke-keys"
Nov 02 13:58:35.995747 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:35.995729    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47c2db8-393e-450f-8dad-28e3cddb860d" containerName="revoke-keys"
Nov 02 13:58:36.001359 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.001326    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.061047 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.061019    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"]
Nov 02 13:58:36.100062 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:36.100032    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd49051_1365_4f49_83bd_26765a94eaac.slice/crio-935995399eb4be1fb753d5e335a10b105ee66208279cb7e3e32aa1e110be40ea WatchSource:0}: Error finding container 935995399eb4be1fb753d5e335a10b105ee66208279cb7e3e32aa1e110be40ea: Status 404 returned error can't find the container with id 935995399eb4be1fb753d5e335a10b105ee66208279cb7e3e32aa1e110be40ea
Nov 02 13:58:36.109667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.109645    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"]
Nov 02 13:58:36.114125 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114103    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114229    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45dfa51-e45e-4497-929c-4fbb9265331f-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114270    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114399 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114381    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwxs\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-kube-api-access-vgwxs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114429    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114515 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114503    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114605 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114582    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114659    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.114729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.114714    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.148776 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.144665    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"]
Nov 02 13:58:36.151677 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.151658    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.180084 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.178930    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"]
Nov 02 13:58:36.216761 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.216724    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.216952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.216799    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.216952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.216860    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.216952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.216905    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.217124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.216985    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.217124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217054    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.217124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217089    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.217245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217121    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.217323 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217303    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-data\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.217383 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217315    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c36986-d034-4084-a190-9a6aad2c8ef0-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.217432 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217415    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.217486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217462    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45dfa51-e45e-4497-929c-4fbb9265331f-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.217749 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217728    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.217871 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217852    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwxs\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-kube-api-access-vgwxs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.217997 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217900    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89f2v\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-kube-api-access-89f2v\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.217997 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.217962    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.218109 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.218033    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45dfa51-e45e-4497-929c-4fbb9265331f-istiod-ca-cert\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.218319 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.218302    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.218376 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.218306    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-certs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.218433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.218376    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.218486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.218432    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.218486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.218462    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.218747 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.218726    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-credential-socket\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.219385 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.219366    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-envoy\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.219737 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.219716    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-podinfo\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.230415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.230386    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwxs\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-kube-api-access-vgwxs\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.230415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.230407    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-token\") pod \"e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") " pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.323583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.322857    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324121    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324178    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324273    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c36986-d034-4084-a190-9a6aad2c8ef0-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324327    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324467    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89f2v\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-kube-api-access-89f2v\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324531    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324638    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324686    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.324723    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.325283    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c36986-d034-4084-a190-9a6aad2c8ef0-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.328586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.325376    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.352454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.351316    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.352454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.351620    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.352454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.351885    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.381079 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.381013    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89f2v\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-kube-api-access-89f2v\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.381865 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.381808    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.384472 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.384414    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.385110 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.385074    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.474323 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.472327    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf" event={"ID":"ccd49051-1365-4f49-83bd-26765a94eaac","Type":"ContainerStarted","Data":"935995399eb4be1fb753d5e335a10b105ee66208279cb7e3e32aa1e110be40ea"}
Nov 02 13:58:36.476615 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.476113    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5" event={"ID":"a47c2db8-393e-450f-8dad-28e3cddb860d","Type":"ContainerDied","Data":"134a49f6b80797ea3ebebccfb0a8c4c65e4de03fa26e4803a1623ed731a713cd"}
Nov 02 13:58:36.476615 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.476157    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"
Nov 02 13:58:36.476615 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.476164    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="134a49f6b80797ea3ebebccfb0a8c4c65e4de03fa26e4803a1623ed731a713cd"
Nov 02 13:58:36.501942 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.487943    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:36.683171 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:36.683090    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45dfa51_e45e_4497_929c_4fbb9265331f.slice/crio-ff4d923f1abfff09bb408848bf496bfa71ecc38434de3359d8e5330b129a9575 WatchSource:0}: Error finding container ff4d923f1abfff09bb408848bf496bfa71ecc38434de3359d8e5330b129a9575: Status 404 returned error can't find the container with id ff4d923f1abfff09bb408848bf496bfa71ecc38434de3359d8e5330b129a9575
Nov 02 13:58:36.684102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.684073    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"]
Nov 02 13:58:36.686345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.686242    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:36.686345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.686323    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:36.686511 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.686364    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:36.794839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.794807    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"]
Nov 02 13:58:36.796339 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:36.796306    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c36986_d034_4084_a190_9a6aad2c8ef0.slice/crio-6e6fcbfffdc22d394240bf5da657fa445f1d56ae6e592672982bec7e8b45660b WatchSource:0}: Error finding container 6e6fcbfffdc22d394240bf5da657fa445f1d56ae6e592672982bec7e8b45660b: Status 404 returned error can't find the container with id 6e6fcbfffdc22d394240bf5da657fa445f1d56ae6e592672982bec7e8b45660b
Nov 02 13:58:36.800424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.800390    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:36.800553 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.800459    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:36.800553 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:36.800488    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:58:37.485681 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.485589    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" event={"ID":"60c36986-d034-4084-a190-9a6aad2c8ef0","Type":"ContainerStarted","Data":"0af59780dcf0db223170c582010c2306fdbab2dbd3ff5ab001e5ab7e54f14bcf"}
Nov 02 13:58:37.485869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.485694    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" event={"ID":"60c36986-d034-4084-a190-9a6aad2c8ef0","Type":"ContainerStarted","Data":"6e6fcbfffdc22d394240bf5da657fa445f1d56ae6e592672982bec7e8b45660b"}
Nov 02 13:58:37.487505 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.487481    5572 generic.go:356] "Generic (PLEG): container finished" podID="ccd49051-1365-4f49-83bd-26765a94eaac" containerID="85a36edcca6848f05a15accdfec1742903cd6dd92873681976004015549f899b" exitCode=0
Nov 02 13:58:37.487665 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.487574    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf" event={"ID":"ccd49051-1365-4f49-83bd-26765a94eaac","Type":"ContainerDied","Data":"85a36edcca6848f05a15accdfec1742903cd6dd92873681976004015549f899b"}
Nov 02 13:58:37.488320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.488302    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:37.489455 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.489430    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" event={"ID":"d45dfa51-e45e-4497-929c-4fbb9265331f","Type":"ContainerStarted","Data":"29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb"}
Nov 02 13:58:37.489536 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.489471    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" event={"ID":"d45dfa51-e45e-4497-929c-4fbb9265331f","Type":"ContainerStarted","Data":"ff4d923f1abfff09bb408848bf496bfa71ecc38434de3359d8e5330b129a9575"}
Nov 02 13:58:37.490043 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.490021    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 13:58:37.490148 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.490075    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 13:58:37.530188 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.530139    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podStartSLOduration=1.530125172 podStartE2EDuration="1.530125172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:37.524995182 +0000 UTC m=+894.103084049" watchObservedRunningTime="2026-11-02 13:58:37.530125172 +0000 UTC m=+894.108214116"
Nov 02 13:58:37.616488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:37.616420    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podStartSLOduration=2.616396547 podStartE2EDuration="2.616396547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:37.611103723 +0000 UTC m=+894.189192591" watchObservedRunningTime="2026-11-02 13:58:37.616396547 +0000 UTC m=+894.194485415"
Nov 02 13:58:38.329384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:38.329347    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:38.330090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:38.330064    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 13:58:38.330254 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:38.330229    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 13:58:38.515583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:38.504483    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 13:58:38.515583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:38.504544    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 13:58:38.866066 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:38.866040    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:39.043055 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.042952    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd49051-1365-4f49-83bd-26765a94eaac-kube-api-access-jms4q\" (UniqueName: \"kubernetes.io/projected/ccd49051-1365-4f49-83bd-26765a94eaac-kube-api-access-jms4q\") pod \"ccd49051-1365-4f49-83bd-26765a94eaac\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") "
Nov 02 13:58:39.043225 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.043100    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-openshift-ca\") pod \"ccd49051-1365-4f49-83bd-26765a94eaac\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") "
Nov 02 13:58:39.043327 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.043310    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-xks-ca\") pod \"ccd49051-1365-4f49-83bd-26765a94eaac\" (UID: \"ccd49051-1365-4f49-83bd-26765a94eaac\") "
Nov 02 13:58:39.043664 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.043636    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-openshift-ca" pod "ccd49051-1365-4f49-83bd-26765a94eaac" (UID: "ccd49051-1365-4f49-83bd-26765a94eaac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:39.044581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.044550    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:39.045765 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.045727    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-xks-ca" pod "ccd49051-1365-4f49-83bd-26765a94eaac" (UID: "ccd49051-1365-4f49-83bd-26765a94eaac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:39.045888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.045787    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd49051-1365-4f49-83bd-26765a94eaac-kube-api-access-jms4q" pod "ccd49051-1365-4f49-83bd-26765a94eaac" (UID: "ccd49051-1365-4f49-83bd-26765a94eaac"). InnerVolumeSpecName "kube-api-access-jms4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:39.146136 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.146090    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd49051-1365-4f49-83bd-26765a94eaac-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:39.146136 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.146131    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jms4q\" (UniqueName: \"kubernetes.io/projected/ccd49051-1365-4f49-83bd-26765a94eaac-kube-api-access-jms4q\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:39.326026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.325989    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 13:58:39.326798 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.326359    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 13:58:39.507594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.496707    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 13:58:39.507594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.506665    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 13:58:39.543983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.543952    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"
Nov 02 13:58:39.549320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.549289    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf" event={"ID":"ccd49051-1365-4f49-83bd-26765a94eaac","Type":"ContainerDied","Data":"935995399eb4be1fb753d5e335a10b105ee66208279cb7e3e32aa1e110be40ea"}
Nov 02 13:58:39.549501 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:39.549329    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="935995399eb4be1fb753d5e335a10b105ee66208279cb7e3e32aa1e110be40ea"
Nov 02 13:58:40.331072 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:40.331035    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 13:58:40.331382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:40.331348    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 13:58:40.503316 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:40.503274    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 13:58:40.504128 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:40.504093    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 13:58:41.335583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:41.332026    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 13:58:41.335583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:41.332095    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 13:58:41.501591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:41.491025    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 13:58:41.501591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:41.491107    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 13:58:42.328585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:42.326962    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 13:58:42.328585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:42.327025    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 13:58:42.502586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:42.490874    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 13:58:42.502586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:42.490933    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 13:58:43.333028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:43.332992    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 13:58:43.333284 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:43.333258    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 13:58:43.506612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:43.491753    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 13:58:43.506612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:43.491811    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 13:58:44.326065 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:44.326026    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 13:58:44.326332 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:44.326080    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 13:58:45.285139 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.285049    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"]
Nov 02 13:58:45.294405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.294371    5572 cpu_manager.go:414] "containerMap: removing container" podUID="ccd49051-1365-4f49-83bd-26765a94eaac" containerName="revoke-keys"
Nov 02 13:58:45.294405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.294403    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd49051-1365-4f49-83bd-26765a94eaac" containerName="revoke-keys"
Nov 02 13:58:45.294828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.294739    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd49051-1365-4f49-83bd-26765a94eaac" containerName="revoke-keys"
Nov 02 13:58:45.307049 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.307023    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:45.313901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.310739    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.314484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.313976    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"]
Nov 02 13:58:45.314674 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.314537    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:45.336648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.332941    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 13:58:45.336648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.333894    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:45.336648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.335406    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:58:45.407951 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.407907    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59jl\" (UniqueName: \"kubernetes.io/projected/98bb0d6b-d72c-4b36-b793-3b1ec339f869-kube-api-access-f59jl\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.408157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.408128    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.408313 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.408294    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.456842 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.456813    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:45.459136 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.459114    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:45.463896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.463873    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 13:58:45.510425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.510383    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.510624 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.510523    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.511396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.510801    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59jl\" (UniqueName: \"kubernetes.io/projected/98bb0d6b-d72c-4b36-b793-3b1ec339f869-kube-api-access-f59jl\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.511396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.511335    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.513055 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.513029    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.528607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.528546    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59jl\" (UniqueName: \"kubernetes.io/projected/98bb0d6b-d72c-4b36-b793-3b1ec339f869-kube-api-access-f59jl\") pod \"maas-api-revoke-sub-e0699e85-kdwb7\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:45.671421 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:45.671386    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:46.476957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:46.476856    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"]
Nov 02 13:58:46.506760 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:58:46.487294    5572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bb0d6b_d72c_4b36_b793_3b1ec339f869.slice/crio-0c3e305baee325ba2216c4ef7e4af787e8d9dd1dd496d40fd2151bcec26a3e5e\": RecentStats: unable to find data in memory cache]"
Nov 02 13:58:46.533594 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:58:46.507658    5572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bb0d6b_d72c_4b36_b793_3b1ec339f869.slice/crio-0c3e305baee325ba2216c4ef7e4af787e8d9dd1dd496d40fd2151bcec26a3e5e\": RecentStats: unable to find data in memory cache]"
Nov 02 13:58:46.836591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:46.814498    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:46.887583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:46.883396    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded"
Nov 02 13:58:46.993599 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:46.929285    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:46.993599 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:46.986535    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.006600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.001063    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7" event={"ID":"98bb0d6b-d72c-4b36-b793-3b1ec339f869","Type":"ContainerStarted","Data":"0c3e305baee325ba2216c4ef7e4af787e8d9dd1dd496d40fd2151bcec26a3e5e"}
Nov 02 13:58:47.178585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.156501    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.211943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.211899    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.309593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.308718    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:58:47.326582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.322268    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.326778 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.326646    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.498590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.497727    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.598627 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.592874    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.742759 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.742696    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:58:47.767999 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.766694    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 13:58:47.767999 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.767160    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 13:58:47.816754 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.816679    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:58:47.819095 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.819071    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:58:47.821713 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.821684    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.050639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.990666    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.050639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:47.991488    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:58:48.150584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.149725    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.150584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.150146    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.175587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.161301    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.200588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.189654    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.200588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.190373    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:58:48.237917 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.215435    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.237917 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.215941    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:58:48.421411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.348135    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.421411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.384167    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:48.421411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.394338    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.421411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.396500    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:58:48.566189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.500822    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.566189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.503242    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:58:48.566189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.506311    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.566189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.509053    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:58:48.594931 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.594796    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.596503 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.596479    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:58:48.648886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.648758    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7" event={"ID":"98bb0d6b-d72c-4b36-b793-3b1ec339f869","Type":"ContainerStarted","Data":"7bb7c4ee08cb2d85079238d048a12ee0cb885bacdb4901b498fdaff0b78ae9f7"}
Nov 02 13:58:48.717479 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.715594    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.750931 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.750832    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.774625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.758886    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:58:48.775036 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.774982    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:48.821503 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.821388    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:58:48.852724 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.850604    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.869592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.852972    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:58:48.869592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:48.853317    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:58:49.033686 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.033634    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.202770 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.202661    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.230830 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.229152    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.231115 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.231081    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.258587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.232961    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:58:49.335643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.245713    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:58:49.335979 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.335959    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:58:49.391586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.386900    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 02 13:58:49.396155 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.396115    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.515189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.513776    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.515189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.514176    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 02 13:58:49.515189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.514238    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:58:49.608781 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.608721    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.763584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.755501    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:58:49.811699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.802708    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.863080 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.863028    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.877690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.850866    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:49.958359 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:49.927866    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:50.151073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.149225    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:58:50.171617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.151282    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.199824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.189765    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7" podStartSLOduration=5.189746222 podStartE2EDuration="5.189746222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:48.939541696 +0000 UTC m=+905.517630562" watchObservedRunningTime="2026-11-02 13:58:50.189746222 +0000 UTC m=+906.767835090"
Nov 02 13:58:50.258940 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.258891    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.259576 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.259522    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:58:50.286590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.272231    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.337306 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.337259    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:58:50.387829 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.387788    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 02 13:58:50.408434 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.407911    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.450890 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.450851    5572 generic.go:356] "Generic (PLEG): container finished" podID="98bb0d6b-d72c-4b36-b793-3b1ec339f869" containerID="7bb7c4ee08cb2d85079238d048a12ee0cb885bacdb4901b498fdaff0b78ae9f7" exitCode=0
Nov 02 13:58:50.451382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.451357    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7" event={"ID":"98bb0d6b-d72c-4b36-b793-3b1ec339f869","Type":"ContainerDied","Data":"7bb7c4ee08cb2d85079238d048a12ee0cb885bacdb4901b498fdaff0b78ae9f7"}
Nov 02 13:58:50.526960 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.502676    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.527606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.527314    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.528242 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.527935    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:58:50.657090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.655674    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:50.657090 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.655741    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.719633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.717887    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:50.719633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.717954    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.756608 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.756546    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.779769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.778670    5572 patch_prober.go:29] interesting pod/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:50.779769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.778736    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.809072 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.809029    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:58:50.819583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.812911    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 13:58:50.868926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.867657    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.883927 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:50.883884    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.161727 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.161674    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:58:51.281993 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.281927    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:58:51.283788 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.283745    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.295579 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.295527    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.343586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.340535    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.390156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.389345    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.425857 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.425756    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.428551 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.428502    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.540301 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.540236    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.544590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.541380    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.807758 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.802819    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.878362 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.828961    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.878362 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.856694    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 13:58:51.910625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.898738    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:51.910625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:51.900923    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:58:52.119694 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:58:52.099971    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:58:52.121949 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.121916    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:52.122242 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.122197    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.162396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.162351    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.283641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.283586    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 13:58:52.284361 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.284327    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:58:52.314744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.314693    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.344201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.344123    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:58:52.434646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.430693    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.434646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.431151    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 02 13:58:52.543385 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.541365    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:58:52.545299 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.544108    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.676598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.674641    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.805196 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.803895    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.822442 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.821998    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:52.822442 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.822059    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.837728 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.831832    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.864646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.863915    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.904439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.902662    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.904439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:52.903043    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:58:53.244142 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.221466    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:58:53.266966 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.266936    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:58:53.267301 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.267285    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:58:53.313502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.283698    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:58:53.313502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.296904    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:58:53.331581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.326240    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:53.333037 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.333002    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:58:53.366255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.366212    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:53.400903 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.399939    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:58:53.484126 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.470695    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 02 13:58:53.635748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.635479    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:58:53.661661 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.661626    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:53.738365 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.738337    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:58:53.738635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.738619    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:58:53.738768 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.738758    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:58:53.738875 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.738866    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:58:53.738993 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.738985    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:58:53.739091 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.739083    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:58:53.739195 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.739187    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:58:53.739289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.739281    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:58:53.739380 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.739373    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:58:53.747114 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.747088    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:58:53.755377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.754504    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:58:53.756201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.756181    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:53.860074 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.860038    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-xks-ca\") pod \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") "
Nov 02 13:58:53.860540 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.860184    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bb0d6b-d72c-4b36-b793-3b1ec339f869-kube-api-access-f59jl\" (UniqueName: \"kubernetes.io/projected/98bb0d6b-d72c-4b36-b793-3b1ec339f869-kube-api-access-f59jl\") pod \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") "
Nov 02 13:58:53.860540 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.860288    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-openshift-ca\") pod \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\" (UID: \"98bb0d6b-d72c-4b36-b793-3b1ec339f869\") "
Nov 02 13:58:53.860880 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.860807    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-openshift-ca" pod "98bb0d6b-d72c-4b36-b793-3b1ec339f869" (UID: "98bb0d6b-d72c-4b36-b793-3b1ec339f869"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:53.861471 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.861449    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:53.861996 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.861968    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7" event={"ID":"98bb0d6b-d72c-4b36-b793-3b1ec339f869","Type":"ContainerDied","Data":"0c3e305baee325ba2216c4ef7e4af787e8d9dd1dd496d40fd2151bcec26a3e5e"}
Nov 02 13:58:53.862087 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.862008    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"
Nov 02 13:58:53.862087 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.862027    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c3e305baee325ba2216c4ef7e4af787e8d9dd1dd496d40fd2151bcec26a3e5e"
Nov 02 13:58:53.862440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.862419    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-xks-ca" pod "98bb0d6b-d72c-4b36-b793-3b1ec339f869" (UID: "98bb0d6b-d72c-4b36-b793-3b1ec339f869"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:53.862634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.862546    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bb0d6b-d72c-4b36-b793-3b1ec339f869-kube-api-access-f59jl" pod "98bb0d6b-d72c-4b36-b793-3b1ec339f869" (UID: "98bb0d6b-d72c-4b36-b793-3b1ec339f869"). InnerVolumeSpecName "kube-api-access-f59jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:53.962995 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.962903    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98bb0d6b-d72c-4b36-b793-3b1ec339f869-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:53.963203 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:53.963181    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59jl\" (UniqueName: \"kubernetes.io/projected/98bb0d6b-d72c-4b36-b793-3b1ec339f869-kube-api-access-f59jl\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:54.348351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.348319    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"]
Nov 02 13:58:54.349161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.349142    5572 cpu_manager.go:414] "containerMap: removing container" podUID="98bb0d6b-d72c-4b36-b793-3b1ec339f869" containerName="revoke-keys"
Nov 02 13:58:54.349161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.349161    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bb0d6b-d72c-4b36-b793-3b1ec339f869" containerName="revoke-keys"
Nov 02 13:58:54.349357 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.349299    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bb0d6b-d72c-4b36-b793-3b1ec339f869" containerName="revoke-keys"
Nov 02 13:58:54.353640 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.353617    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.372243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.372222    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:54.372531 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.372514    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:58:54.394434 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.394403    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"]
Nov 02 13:58:54.472913 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.472876    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.472913 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.472916    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.473265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.473244    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hx8\" (UniqueName: \"kubernetes.io/projected/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-kube-api-access-d7hx8\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.575226 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.575195    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hx8\" (UniqueName: \"kubernetes.io/projected/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-kube-api-access-d7hx8\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.575406 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.575255    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.575406 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.575284    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.576081 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.576057    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.577996 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.577976    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.588252 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.588232    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hx8\" (UniqueName: \"kubernetes.io/projected/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-kube-api-access-d7hx8\") pod \"maas-api-revoke-sub-5b03ef32-wfzng\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:54.682637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:54.677854    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:55.117855 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:55.117808    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265c9673_ae2d_4f7d_b93e_bc35eae5b5c1.slice/crio-1a81a2c21fb09bf2e25501e58772a2c50c48a6874b4f0a691a9c2388342c1bdc WatchSource:0}: Error finding container 1a81a2c21fb09bf2e25501e58772a2c50c48a6874b4f0a691a9c2388342c1bdc: Status 404 returned error can't find the container with id 1a81a2c21fb09bf2e25501e58772a2c50c48a6874b4f0a691a9c2388342c1bdc
Nov 02 13:58:55.118519 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:55.118487    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"]
Nov 02 13:58:55.907229 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:55.907125    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng" event={"ID":"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1","Type":"ContainerStarted","Data":"b542b519cafa15a4bfd75ddf14a8f0923c059de8a0fed4286fd73d1eb0eb1f36"}
Nov 02 13:58:55.907399 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:55.907229    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng" event={"ID":"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1","Type":"ContainerStarted","Data":"1a81a2c21fb09bf2e25501e58772a2c50c48a6874b4f0a691a9c2388342c1bdc"}
Nov 02 13:58:55.930028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:55.929971    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng" podStartSLOduration=1.929958281 podStartE2EDuration="1.929958281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:55.927372257 +0000 UTC m=+912.505461127" watchObservedRunningTime="2026-11-02 13:58:55.929958281 +0000 UTC m=+912.508047148"
Nov 02 13:58:56.917490 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:56.917457    5572 generic.go:356] "Generic (PLEG): container finished" podID="265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" containerID="b542b519cafa15a4bfd75ddf14a8f0923c059de8a0fed4286fd73d1eb0eb1f36" exitCode=0
Nov 02 13:58:56.917976 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:56.917534    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng" event={"ID":"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1","Type":"ContainerDied","Data":"b542b519cafa15a4bfd75ddf14a8f0923c059de8a0fed4286fd73d1eb0eb1f36"}
Nov 02 13:58:57.088909 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.088873    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"]
Nov 02 13:58:57.093578 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.093541    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.116762 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.116727    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"]
Nov 02 13:58:57.141744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.141707    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.141909 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.141838    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.142067 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.142040    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhfs\" (UniqueName: \"kubernetes.io/projected/c6e6076b-990e-4c31-a445-7f35cf72ecdb-kube-api-access-kdhfs\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.243827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.243733    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.243827 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.243795    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.244071 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.243884    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhfs\" (UniqueName: \"kubernetes.io/projected/c6e6076b-990e-4c31-a445-7f35cf72ecdb-kube-api-access-kdhfs\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.244519 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.244500    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.246293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.246269    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.259772 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.259748    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhfs\" (UniqueName: \"kubernetes.io/projected/c6e6076b-990e-4c31-a445-7f35cf72ecdb-kube-api-access-kdhfs\") pod \"maas-api-revoke-sub-a8a2a5c4-mlmss\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.408283 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.408251    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:58:57.611518 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.611488    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"]
Nov 02 13:58:57.616280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.616260    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.629663 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.629639    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"]
Nov 02 13:58:57.650676 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.650645    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.650864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.650684    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6fmc\" (UniqueName: \"kubernetes.io/projected/7a003a3f-c510-442b-8f3a-ff22263e662d-kube-api-access-l6fmc\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.650864 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.650797    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.752868 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.752833    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.753057 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.752902    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6fmc\" (UniqueName: \"kubernetes.io/projected/7a003a3f-c510-442b-8f3a-ff22263e662d-kube-api-access-l6fmc\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.753057 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.752993    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.754508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.753958    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.757317 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.757294    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.769428 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.769379    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"]
Nov 02 13:58:57.783448 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.783376    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6fmc\" (UniqueName: \"kubernetes.io/projected/7a003a3f-c510-442b-8f3a-ff22263e662d-kube-api-access-l6fmc\") pod \"maas-api-revoke-sub-97339148-qwgb2\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:57.926206 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.926172    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss" event={"ID":"c6e6076b-990e-4c31-a445-7f35cf72ecdb","Type":"ContainerStarted","Data":"03672d11027eb749a40d3d9a78c27c852b45ed124cc00a5f7b6008d9b7427322"}
Nov 02 13:58:57.932844 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:57.932823    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:58:58.106940 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.106916    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:58.130510 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.130442    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"]
Nov 02 13:58:58.131171 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.131156    5572 cpu_manager.go:414] "containerMap: removing container" podUID="265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" containerName="revoke-keys"
Nov 02 13:58:58.131212 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.131173    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" containerName="revoke-keys"
Nov 02 13:58:58.131305 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.131297    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" containerName="revoke-keys"
Nov 02 13:58:58.134951 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.134935    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.161643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.161607    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"]
Nov 02 13:58:58.162032 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.162010    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-kube-api-access-d7hx8\" (UniqueName: \"kubernetes.io/projected/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-kube-api-access-d7hx8\") pod \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") "
Nov 02 13:58:58.162113 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.162060    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-xks-ca\") pod \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") "
Nov 02 13:58:58.162289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.162271    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-openshift-ca\") pod \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\" (UID: \"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1\") "
Nov 02 13:58:58.162701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.162671    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-openshift-ca" pod "265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" (UID: "265c9673-ae2d-4f7d-b93e-bc35eae5b5c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:58.163387 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.163361    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.163515 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.163415    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntrh\" (UniqueName: \"kubernetes.io/projected/937a2b67-97c5-4988-a623-3bcb830c8e33-kube-api-access-nntrh\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.163673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.163650    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.163901 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.163880    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:58.164543 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.164515    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-kube-api-access-d7hx8" pod "265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" (UID: "265c9673-ae2d-4f7d-b93e-bc35eae5b5c1"). InnerVolumeSpecName "kube-api-access-d7hx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:58.164680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.164604    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-xks-ca" pod "265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" (UID: "265c9673-ae2d-4f7d-b93e-bc35eae5b5c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:58.230635 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:58.230606    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a003a3f_c510_442b_8f3a_ff22263e662d.slice/crio-9f424a3803eea6f01903da3cd2a468cd05197a2fe1f8d6d0a3a90b2e0fb56b1a WatchSource:0}: Error finding container 9f424a3803eea6f01903da3cd2a468cd05197a2fe1f8d6d0a3a90b2e0fb56b1a: Status 404 returned error can't find the container with id 9f424a3803eea6f01903da3cd2a468cd05197a2fe1f8d6d0a3a90b2e0fb56b1a
Nov 02 13:58:58.232618 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.232592    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"]
Nov 02 13:58:58.269839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.265946    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.269839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.266475    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntrh\" (UniqueName: \"kubernetes.io/projected/937a2b67-97c5-4988-a623-3bcb830c8e33-kube-api-access-nntrh\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.269839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.266629    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.269839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.267052    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hx8\" (UniqueName: \"kubernetes.io/projected/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-kube-api-access-d7hx8\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:58.269839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.267075    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:58:58.269839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.268666    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.269839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.269431    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.279002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.278973    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntrh\" (UniqueName: \"kubernetes.io/projected/937a2b67-97c5-4988-a623-3bcb830c8e33-kube-api-access-nntrh\") pod \"maas-api-revoke-sub-3f822813-w45nz\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.457126 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.457048    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:58:58.789216 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:58:58.789189    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937a2b67_97c5_4988_a623_3bcb830c8e33.slice/crio-9d37e84ee1072e9ecb2b2d97fa95043e104c9a5e38d885b6b24330723678fefd WatchSource:0}: Error finding container 9d37e84ee1072e9ecb2b2d97fa95043e104c9a5e38d885b6b24330723678fefd: Status 404 returned error can't find the container with id 9d37e84ee1072e9ecb2b2d97fa95043e104c9a5e38d885b6b24330723678fefd
Nov 02 13:58:58.789311 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.789217    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"]
Nov 02 13:58:58.935903 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.935867    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2" event={"ID":"7a003a3f-c510-442b-8f3a-ff22263e662d","Type":"ContainerStarted","Data":"3c9ab4f132fc4f3615264cd8ad37954b8ebf1b0e4adb590e6cded51a20e7cd18"}
Nov 02 13:58:58.936415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.935918    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2" event={"ID":"7a003a3f-c510-442b-8f3a-ff22263e662d","Type":"ContainerStarted","Data":"9f424a3803eea6f01903da3cd2a468cd05197a2fe1f8d6d0a3a90b2e0fb56b1a"}
Nov 02 13:58:58.937631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.937606    5572 generic.go:356] "Generic (PLEG): container finished" podID="c6e6076b-990e-4c31-a445-7f35cf72ecdb" containerID="c687e6274a6826253a694b13507bcace005d3a8a1ebf29cf40e91c2cc8a9faa8" exitCode=0
Nov 02 13:58:58.937754 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.937677    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss" event={"ID":"c6e6076b-990e-4c31-a445-7f35cf72ecdb","Type":"ContainerDied","Data":"c687e6274a6826253a694b13507bcace005d3a8a1ebf29cf40e91c2cc8a9faa8"}
Nov 02 13:58:58.938910 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.938895    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"
Nov 02 13:58:58.939000 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.938913    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng" event={"ID":"265c9673-ae2d-4f7d-b93e-bc35eae5b5c1","Type":"ContainerDied","Data":"1a81a2c21fb09bf2e25501e58772a2c50c48a6874b4f0a691a9c2388342c1bdc"}
Nov 02 13:58:58.939000 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.938946    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a81a2c21fb09bf2e25501e58772a2c50c48a6874b4f0a691a9c2388342c1bdc"
Nov 02 13:58:58.940356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.940331    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz" event={"ID":"937a2b67-97c5-4988-a623-3bcb830c8e33","Type":"ContainerStarted","Data":"9d37e84ee1072e9ecb2b2d97fa95043e104c9a5e38d885b6b24330723678fefd"}
Nov 02 13:58:58.960368 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:58.960325    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2" podStartSLOduration=1.9603135470000002 podStartE2EDuration="1.960313547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:58.957842628 +0000 UTC m=+915.535931492" watchObservedRunningTime="2026-11-02 13:58:58.960313547 +0000 UTC m=+915.538402414"
Nov 02 13:58:59.481756 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.481726    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"]
Nov 02 13:58:59.486311 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.486282    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.503847 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.503822    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"]
Nov 02 13:58:59.619375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.619332    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.619909 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.619883    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.620069 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.620050    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-kube-api-access-lbch7\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.721532 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.721499    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-kube-api-access-lbch7\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.721744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.721631    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.721835 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.721822    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.722281 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.722255    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.724577 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.724539    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.734390 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.734368    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-kube-api-access-lbch7\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.805551 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.805523    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:58:59.953720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.953617    5572 generic.go:356] "Generic (PLEG): container finished" podID="7a003a3f-c510-442b-8f3a-ff22263e662d" containerID="3c9ab4f132fc4f3615264cd8ad37954b8ebf1b0e4adb590e6cded51a20e7cd18" exitCode=0
Nov 02 13:58:59.953720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.953675    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2" event={"ID":"7a003a3f-c510-442b-8f3a-ff22263e662d","Type":"ContainerDied","Data":"3c9ab4f132fc4f3615264cd8ad37954b8ebf1b0e4adb590e6cded51a20e7cd18"}
Nov 02 13:58:59.955873 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.955847    5572 generic.go:356] "Generic (PLEG): container finished" podID="937a2b67-97c5-4988-a623-3bcb830c8e33" containerID="97a14b46252e9621b94bbd477e0f00c3df6953ecdd256a8d66252b731e408872" exitCode=0
Nov 02 13:58:59.956011 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:58:59.955998    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz" event={"ID":"937a2b67-97c5-4988-a623-3bcb830c8e33","Type":"ContainerDied","Data":"97a14b46252e9621b94bbd477e0f00c3df6953ecdd256a8d66252b731e408872"}
Nov 02 13:59:00.108471 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:00.108438    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483ae1b4_7f0e_4b7d_8ec0_b55663fc67f9.slice/crio-36e1008b87f480025aa24d3259eea52600ab2736ffe76ad09f81ff66202e9bb3 WatchSource:0}: Error finding container 36e1008b87f480025aa24d3259eea52600ab2736ffe76ad09f81ff66202e9bb3: Status 404 returned error can't find the container with id 36e1008b87f480025aa24d3259eea52600ab2736ffe76ad09f81ff66202e9bb3
Nov 02 13:59:00.110764 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.110738    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"]
Nov 02 13:59:00.116983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.116961    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:59:00.237041 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.233336    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-openshift-ca\") pod \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") "
Nov 02 13:59:00.243785 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.243743    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-xks-ca\") pod \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") "
Nov 02 13:59:00.244030 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.244008    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e6076b-990e-4c31-a445-7f35cf72ecdb-kube-api-access-kdhfs\" (UniqueName: \"kubernetes.io/projected/c6e6076b-990e-4c31-a445-7f35cf72ecdb-kube-api-access-kdhfs\") pod \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\" (UID: \"c6e6076b-990e-4c31-a445-7f35cf72ecdb\") "
Nov 02 13:59:00.245084 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.245055    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-openshift-ca" pod "c6e6076b-990e-4c31-a445-7f35cf72ecdb" (UID: "c6e6076b-990e-4c31-a445-7f35cf72ecdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:00.247635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.247613    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:00.292237 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.287717    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-xks-ca" pod "c6e6076b-990e-4c31-a445-7f35cf72ecdb" (UID: "c6e6076b-990e-4c31-a445-7f35cf72ecdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:00.292392 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.292336    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e6076b-990e-4c31-a445-7f35cf72ecdb-kube-api-access-kdhfs" pod "c6e6076b-990e-4c31-a445-7f35cf72ecdb" (UID: "c6e6076b-990e-4c31-a445-7f35cf72ecdb"). InnerVolumeSpecName "kube-api-access-kdhfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:00.350340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.350309    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6e6076b-990e-4c31-a445-7f35cf72ecdb-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:00.350340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.350335    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdhfs\" (UniqueName: \"kubernetes.io/projected/c6e6076b-990e-4c31-a445-7f35cf72ecdb-kube-api-access-kdhfs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:00.966510 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.966480    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"
Nov 02 13:59:00.966510 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.966492    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss" event={"ID":"c6e6076b-990e-4c31-a445-7f35cf72ecdb","Type":"ContainerDied","Data":"03672d11027eb749a40d3d9a78c27c852b45ed124cc00a5f7b6008d9b7427322"}
Nov 02 13:59:00.967054 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.966541    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03672d11027eb749a40d3d9a78c27c852b45ed124cc00a5f7b6008d9b7427322"
Nov 02 13:59:00.968608 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.968550    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d" event={"ID":"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9","Type":"ContainerStarted","Data":"09eb7441d1c347dd480cee277c182c8dc66d8ab382a8dd53900a6664dfda3589"}
Nov 02 13:59:00.968705 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:00.968640    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d" event={"ID":"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9","Type":"ContainerStarted","Data":"36e1008b87f480025aa24d3259eea52600ab2736ffe76ad09f81ff66202e9bb3"}
Nov 02 13:59:01.042606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.042526    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d" podStartSLOduration=2.042512147 podStartE2EDuration="2.042512147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:01.035655384 +0000 UTC m=+917.613744251" watchObservedRunningTime="2026-11-02 13:59:01.042512147 +0000 UTC m=+917.620601014"
Nov 02 13:59:01.209372 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.209345    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:59:01.213408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.213386    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:59:01.366725 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.366692    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-openshift-ca\") pod \"937a2b67-97c5-4988-a623-3bcb830c8e33\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") "
Nov 02 13:59:01.366932 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.366755    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-openshift-ca\") pod \"7a003a3f-c510-442b-8f3a-ff22263e662d\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") "
Nov 02 13:59:01.366932 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.366807    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-xks-ca\") pod \"7a003a3f-c510-442b-8f3a-ff22263e662d\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") "
Nov 02 13:59:01.367080 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.367060    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a003a3f-c510-442b-8f3a-ff22263e662d-kube-api-access-l6fmc\" (UniqueName: \"kubernetes.io/projected/7a003a3f-c510-442b-8f3a-ff22263e662d-kube-api-access-l6fmc\") pod \"7a003a3f-c510-442b-8f3a-ff22263e662d\" (UID: \"7a003a3f-c510-442b-8f3a-ff22263e662d\") "
Nov 02 13:59:01.367142 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.367115    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-openshift-ca" pod "937a2b67-97c5-4988-a623-3bcb830c8e33" (UID: "937a2b67-97c5-4988-a623-3bcb830c8e33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:01.367142 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.367123    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-openshift-ca" pod "7a003a3f-c510-442b-8f3a-ff22263e662d" (UID: "7a003a3f-c510-442b-8f3a-ff22263e662d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:01.367222 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.367171    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937a2b67-97c5-4988-a623-3bcb830c8e33-kube-api-access-nntrh\" (UniqueName: \"kubernetes.io/projected/937a2b67-97c5-4988-a623-3bcb830c8e33-kube-api-access-nntrh\") pod \"937a2b67-97c5-4988-a623-3bcb830c8e33\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") "
Nov 02 13:59:01.367222 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.367198    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-xks-ca\") pod \"937a2b67-97c5-4988-a623-3bcb830c8e33\" (UID: \"937a2b67-97c5-4988-a623-3bcb830c8e33\") "
Nov 02 13:59:01.367868 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.367845    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:01.367868 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.367866    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:01.369255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.369231    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a003a3f-c510-442b-8f3a-ff22263e662d-kube-api-access-l6fmc" pod "7a003a3f-c510-442b-8f3a-ff22263e662d" (UID: "7a003a3f-c510-442b-8f3a-ff22263e662d"). InnerVolumeSpecName "kube-api-access-l6fmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:01.369255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.369236    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-xks-ca" pod "7a003a3f-c510-442b-8f3a-ff22263e662d" (UID: "7a003a3f-c510-442b-8f3a-ff22263e662d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:01.369834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.369815    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-xks-ca" pod "937a2b67-97c5-4988-a623-3bcb830c8e33" (UID: "937a2b67-97c5-4988-a623-3bcb830c8e33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:01.369896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.369859    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937a2b67-97c5-4988-a623-3bcb830c8e33-kube-api-access-nntrh" pod "937a2b67-97c5-4988-a623-3bcb830c8e33" (UID: "937a2b67-97c5-4988-a623-3bcb830c8e33"). InnerVolumeSpecName "kube-api-access-nntrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:01.470028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.469993    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a003a3f-c510-442b-8f3a-ff22263e662d-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:01.470028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.470023    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6fmc\" (UniqueName: \"kubernetes.io/projected/7a003a3f-c510-442b-8f3a-ff22263e662d-kube-api-access-l6fmc\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:01.470028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.470036    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntrh\" (UniqueName: \"kubernetes.io/projected/937a2b67-97c5-4988-a623-3bcb830c8e33-kube-api-access-nntrh\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:01.470257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.470050    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937a2b67-97c5-4988-a623-3bcb830c8e33-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:01.978099 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.978065    5572 generic.go:356] "Generic (PLEG): container finished" podID="483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" containerID="09eb7441d1c347dd480cee277c182c8dc66d8ab382a8dd53900a6664dfda3589" exitCode=0
Nov 02 13:59:01.978580 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.978148    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d" event={"ID":"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9","Type":"ContainerDied","Data":"09eb7441d1c347dd480cee277c182c8dc66d8ab382a8dd53900a6664dfda3589"}
Nov 02 13:59:01.979709 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.979691    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"
Nov 02 13:59:01.979830 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.979722    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz" event={"ID":"937a2b67-97c5-4988-a623-3bcb830c8e33","Type":"ContainerDied","Data":"9d37e84ee1072e9ecb2b2d97fa95043e104c9a5e38d885b6b24330723678fefd"}
Nov 02 13:59:01.979830 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.979768    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d37e84ee1072e9ecb2b2d97fa95043e104c9a5e38d885b6b24330723678fefd"
Nov 02 13:59:01.981280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.981265    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"
Nov 02 13:59:01.981280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.981269    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2" event={"ID":"7a003a3f-c510-442b-8f3a-ff22263e662d","Type":"ContainerDied","Data":"9f424a3803eea6f01903da3cd2a468cd05197a2fe1f8d6d0a3a90b2e0fb56b1a"}
Nov 02 13:59:01.981402 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:01.981316    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f424a3803eea6f01903da3cd2a468cd05197a2fe1f8d6d0a3a90b2e0fb56b1a"
Nov 02 13:59:03.165806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.165781    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:59:03.261093 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.261004    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-kube-api-access-lbch7\") pod \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") "
Nov 02 13:59:03.261093 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.261052    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-openshift-ca\") pod \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") "
Nov 02 13:59:03.261326 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.261185    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-xks-ca\") pod \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\" (UID: \"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9\") "
Nov 02 13:59:03.261521 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.261492    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-openshift-ca" pod "483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" (UID: "483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:03.261758 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.261742    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:03.263481 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.263453    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-xks-ca" pod "483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" (UID: "483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:03.263661 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.263484    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-kube-api-access-lbch7" pod "483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" (UID: "483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9"). InnerVolumeSpecName "kube-api-access-lbch7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:03.363356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.363324    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-kube-api-access-lbch7\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:03.363356 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.363353    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:03.999203 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.999177    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"
Nov 02 13:59:03.999372 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.999203    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d" event={"ID":"483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9","Type":"ContainerDied","Data":"36e1008b87f480025aa24d3259eea52600ab2736ffe76ad09f81ff66202e9bb3"}
Nov 02 13:59:03.999372 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:03.999251    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e1008b87f480025aa24d3259eea52600ab2736ffe76ad09f81ff66202e9bb3"
Nov 02 13:59:04.239672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:04.239636    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"]
Nov 02 13:59:04.253711 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:04.253629    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"]
Nov 02 13:59:05.986267 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.986224    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"]
Nov 02 13:59:05.986967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.986948    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c6e6076b-990e-4c31-a445-7f35cf72ecdb" containerName="revoke-keys"
Nov 02 13:59:05.986967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.986967    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e6076b-990e-4c31-a445-7f35cf72ecdb" containerName="revoke-keys"
Nov 02 13:59:05.987104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.986986    5572 cpu_manager.go:414] "containerMap: removing container" podUID="7a003a3f-c510-442b-8f3a-ff22263e662d" containerName="revoke-keys"
Nov 02 13:59:05.987104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.986991    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a003a3f-c510-442b-8f3a-ff22263e662d" containerName="revoke-keys"
Nov 02 13:59:05.987104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.986998    5572 cpu_manager.go:414] "containerMap: removing container" podUID="937a2b67-97c5-4988-a623-3bcb830c8e33" containerName="revoke-keys"
Nov 02 13:59:05.987104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.987004    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937a2b67-97c5-4988-a623-3bcb830c8e33" containerName="revoke-keys"
Nov 02 13:59:05.987104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.987010    5572 cpu_manager.go:414] "containerMap: removing container" podUID="483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" containerName="revoke-keys"
Nov 02 13:59:05.987104 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.987015    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" containerName="revoke-keys"
Nov 02 13:59:05.987384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.987134    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e6076b-990e-4c31-a445-7f35cf72ecdb" containerName="revoke-keys"
Nov 02 13:59:05.987384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.987144    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a003a3f-c510-442b-8f3a-ff22263e662d" containerName="revoke-keys"
Nov 02 13:59:05.987384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.987149    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="937a2b67-97c5-4988-a623-3bcb830c8e33" containerName="revoke-keys"
Nov 02 13:59:05.987384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.987156    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" containerName="revoke-keys"
Nov 02 13:59:05.991579 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:05.991540    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.004269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.004231    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"]
Nov 02 13:59:06.015739 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.015705    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxb72\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-kube-api-access-nxb72\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.015896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.015767    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.015896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.015875    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.016011 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.015957    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.016086 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.016059    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.016201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.016179    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.016343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.016327    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7c381f-262b-47f1-a570-1dd247e0955f-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.016415 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.016401    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.016467 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.016456    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.117824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.117784    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118029 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.117858    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118029 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.117913    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7c381f-262b-47f1-a570-1dd247e0955f-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118029 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.117956    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118029 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.117990    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118256 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118056    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxb72\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-kube-api-access-nxb72\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118256 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118092    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118256 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118184    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118256 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118223    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118363    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-credential-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118396    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-data\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118623 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118600    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-socket\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.118699 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.118665    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-certs\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.119051 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.119033    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7c381f-262b-47f1-a570-1dd247e0955f-istiod-ca-cert\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.120525 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.120502    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-podinfo\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.120814 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.120795    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-envoy\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.132928 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.132899    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-token\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.133549 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.133523    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxb72\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-kube-api-access-nxb72\") pod \"e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") " pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.317201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.317168    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:06.611442 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:06.611407    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7c381f_262b_47f1_a570_1dd247e0955f.slice/crio-e2cd525d4637a984d35afc14a3e1ecad8707006d7a89147d37798d48fd458cd4 WatchSource:0}: Error finding container e2cd525d4637a984d35afc14a3e1ecad8707006d7a89147d37798d48fd458cd4: Status 404 returned error can't find the container with id e2cd525d4637a984d35afc14a3e1ecad8707006d7a89147d37798d48fd458cd4
Nov 02 13:59:06.611777 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.611755    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"]
Nov 02 13:59:06.614174 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.614138    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:59:06.614262 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.614223    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:59:06.614300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:06.614263    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:59:07.060038 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.060004    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" event={"ID":"2f7c381f-262b-47f1-a570-1dd247e0955f","Type":"ContainerStarted","Data":"747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362"}
Nov 02 13:59:07.060544 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.060521    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" event={"ID":"2f7c381f-262b-47f1-a570-1dd247e0955f","Type":"ContainerStarted","Data":"e2cd525d4637a984d35afc14a3e1ecad8707006d7a89147d37798d48fd458cd4"}
Nov 02 13:59:07.097748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.097694    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podStartSLOduration=2.097678987 podStartE2EDuration="2.097678987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:07.094173732 +0000 UTC m=+923.672262600" watchObservedRunningTime="2026-11-02 13:59:07.097678987 +0000 UTC m=+923.675767916"
Nov 02 13:59:07.317886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.317804    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:07.319722 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.319695    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:07.319852 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.319743    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:07.418185 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.418153    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"]
Nov 02 13:59:07.423371 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.423345    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.428337 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.428308    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:07.431338 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.431285    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:59:07.441675 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.441643    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"]
Nov 02 13:59:07.445183 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.445153    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.445417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.445223    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fp52\" (UniqueName: \"kubernetes.io/projected/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-kube-api-access-2fp52\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.445533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.445511    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.548894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.548848    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.549072 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.548936    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fp52\" (UniqueName: \"kubernetes.io/projected/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-kube-api-access-2fp52\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.549172 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.549151    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.552473 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.552440    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.553167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.553140    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.569036 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.568973    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fp52\" (UniqueName: \"kubernetes.io/projected/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-kube-api-access-2fp52\") pod \"maas-api-revoke-sub-79f833d2-xwhwq\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:07.746370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:07.746334    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:08.098968 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:08.097896    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084a1059_63a8_4a2e_b6a0_29c3c7b77bf8.slice/crio-b9c36908c1351d2e04accf0d75202fab3f8f92e51a1df48c896b3853aadfc71e WatchSource:0}: Error finding container b9c36908c1351d2e04accf0d75202fab3f8f92e51a1df48c896b3853aadfc71e: Status 404 returned error can't find the container with id b9c36908c1351d2e04accf0d75202fab3f8f92e51a1df48c896b3853aadfc71e
Nov 02 13:59:08.110423 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.110368    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"]
Nov 02 13:59:08.327584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.320941    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:08.327584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.321005    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:08.453992 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.453956    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"]
Nov 02 13:59:08.463606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.461059    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.463606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.461125    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"]
Nov 02 13:59:08.478716 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.478682    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.478889 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.478871    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgp7w\" (UniqueName: \"kubernetes.io/projected/9866232a-c748-4d81-a74f-1bc790f84bcc-kube-api-access-bgp7w\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.478974 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.478956    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.605748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.580665    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.605748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.580916    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgp7w\" (UniqueName: \"kubernetes.io/projected/9866232a-c748-4d81-a74f-1bc790f84bcc-kube-api-access-bgp7w\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.605748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.580999    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.605748 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.581509    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.608306 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.608276    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgp7w\" (UniqueName: \"kubernetes.io/projected/9866232a-c748-4d81-a74f-1bc790f84bcc-kube-api-access-bgp7w\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.608488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.608378    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded92992-qt2rs\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:08.811411 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:08.811373    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:09.079698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:09.079663    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq" event={"ID":"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8","Type":"ContainerStarted","Data":"aad60d322ad43aabd427370bd6fd49b0a3aff3516555b327969d0ec8ddbe9440"}
Nov 02 13:59:09.079880 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:09.079717    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq" event={"ID":"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8","Type":"ContainerStarted","Data":"b9c36908c1351d2e04accf0d75202fab3f8f92e51a1df48c896b3853aadfc71e"}
Nov 02 13:59:09.113157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:09.113122    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"]
Nov 02 13:59:09.115278 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:09.115252    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9866232a_c748_4d81_a74f_1bc790f84bcc.slice/crio-6bb8611e476eba6d54f991bfa6ab32701c94c7d125e3a51913b557e22a25edf5 WatchSource:0}: Error finding container 6bb8611e476eba6d54f991bfa6ab32701c94c7d125e3a51913b557e22a25edf5: Status 404 returned error can't find the container with id 6bb8611e476eba6d54f991bfa6ab32701c94c7d125e3a51913b557e22a25edf5
Nov 02 13:59:09.118706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:09.118667    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq" podStartSLOduration=2.118654065 podStartE2EDuration="2.118654065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:09.111807656 +0000 UTC m=+925.689896712" watchObservedRunningTime="2026-11-02 13:59:09.118654065 +0000 UTC m=+925.696742932"
Nov 02 13:59:09.318020 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:09.317982    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:09.318176 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:09.318045    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:09.774639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:09.756469    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"]
Nov 02 13:59:10.112586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:10.109422    5572 generic.go:356] "Generic (PLEG): container finished" podID="084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" containerID="aad60d322ad43aabd427370bd6fd49b0a3aff3516555b327969d0ec8ddbe9440" exitCode=0
Nov 02 13:59:10.112586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:10.109642    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq" event={"ID":"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8","Type":"ContainerDied","Data":"aad60d322ad43aabd427370bd6fd49b0a3aff3516555b327969d0ec8ddbe9440"}
Nov 02 13:59:10.144580 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:10.141378    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs" event={"ID":"9866232a-c748-4d81-a74f-1bc790f84bcc","Type":"ContainerStarted","Data":"7471984d5d25b468c49fd9dc57c48fd189a7129ef03a97f8434180b4b4fdd141"}
Nov 02 13:59:10.144580 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:10.141456    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs" event={"ID":"9866232a-c748-4d81-a74f-1bc790f84bcc","Type":"ContainerStarted","Data":"6bb8611e476eba6d54f991bfa6ab32701c94c7d125e3a51913b557e22a25edf5"}
Nov 02 13:59:10.296601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:10.284501    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs" podStartSLOduration=2.284479217 podStartE2EDuration="2.284479217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:10.259315568 +0000 UTC m=+926.837404436" watchObservedRunningTime="2026-11-02 13:59:10.284479217 +0000 UTC m=+926.862568086"
Nov 02 13:59:10.325093 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:10.324025    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:10.325093 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:10.324083    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:11.152088 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.152046    5572 generic.go:356] "Generic (PLEG): container finished" podID="9866232a-c748-4d81-a74f-1bc790f84bcc" containerID="7471984d5d25b468c49fd9dc57c48fd189a7129ef03a97f8434180b4b4fdd141" exitCode=0
Nov 02 13:59:11.152620 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.152125    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs" event={"ID":"9866232a-c748-4d81-a74f-1bc790f84bcc","Type":"ContainerDied","Data":"7471984d5d25b468c49fd9dc57c48fd189a7129ef03a97f8434180b4b4fdd141"}
Nov 02 13:59:11.325896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.325259    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:11.325896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.325323    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:11.638612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.631737    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:11.822444 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.822406    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-kube-api-access-2fp52\" (UniqueName: \"kubernetes.io/projected/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-kube-api-access-2fp52\") pod \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") "
Nov 02 13:59:11.822639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.822529    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-xks-ca\") pod \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") "
Nov 02 13:59:11.822700 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.822658    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-openshift-ca\") pod \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\" (UID: \"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8\") "
Nov 02 13:59:11.823200 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.823172    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-openshift-ca" pod "084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" (UID: "084a1059-63a8-4a2e-b6a0-29c3c7b77bf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:11.823666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.823639    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:11.825778 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.825393    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-kube-api-access-2fp52" pod "084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" (UID: "084a1059-63a8-4a2e-b6a0-29c3c7b77bf8"). InnerVolumeSpecName "kube-api-access-2fp52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:11.826930 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.826907    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-xks-ca" pod "084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" (UID: "084a1059-63a8-4a2e-b6a0-29c3c7b77bf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:11.924701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.924664    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fp52\" (UniqueName: \"kubernetes.io/projected/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-kube-api-access-2fp52\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:11.924701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:11.924695    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:12.161798 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.161717    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"
Nov 02 13:59:12.161798 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.161755    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq" event={"ID":"084a1059-63a8-4a2e-b6a0-29c3c7b77bf8","Type":"ContainerDied","Data":"b9c36908c1351d2e04accf0d75202fab3f8f92e51a1df48c896b3853aadfc71e"}
Nov 02 13:59:12.162228 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.161813    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c36908c1351d2e04accf0d75202fab3f8f92e51a1df48c896b3853aadfc71e"
Nov 02 13:59:12.318213 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.318178    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:12.318386 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.318241    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:12.340967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.340940    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:12.552063 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.552031    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9866232a-c748-4d81-a74f-1bc790f84bcc-kube-api-access-bgp7w\" (UniqueName: \"kubernetes.io/projected/9866232a-c748-4d81-a74f-1bc790f84bcc-kube-api-access-bgp7w\") pod \"9866232a-c748-4d81-a74f-1bc790f84bcc\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") "
Nov 02 13:59:12.552246 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.552110    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-openshift-ca\") pod \"9866232a-c748-4d81-a74f-1bc790f84bcc\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") "
Nov 02 13:59:12.552246 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.552168    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-xks-ca\") pod \"9866232a-c748-4d81-a74f-1bc790f84bcc\" (UID: \"9866232a-c748-4d81-a74f-1bc790f84bcc\") "
Nov 02 13:59:12.555225 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.555195    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-openshift-ca" pod "9866232a-c748-4d81-a74f-1bc790f84bcc" (UID: "9866232a-c748-4d81-a74f-1bc790f84bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:12.565812 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.565758    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9866232a-c748-4d81-a74f-1bc790f84bcc-kube-api-access-bgp7w" pod "9866232a-c748-4d81-a74f-1bc790f84bcc" (UID: "9866232a-c748-4d81-a74f-1bc790f84bcc"). InnerVolumeSpecName "kube-api-access-bgp7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:12.566086 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.566041    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-xks-ca" pod "9866232a-c748-4d81-a74f-1bc790f84bcc" (UID: "9866232a-c748-4d81-a74f-1bc790f84bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:12.654240 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.654205    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgp7w\" (UniqueName: \"kubernetes.io/projected/9866232a-c748-4d81-a74f-1bc790f84bcc-kube-api-access-bgp7w\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:12.654240 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.654236    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:12.654240 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:12.654247    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9866232a-c748-4d81-a74f-1bc790f84bcc-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:13.170720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:13.170691    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"
Nov 02 13:59:13.171106 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:13.170724    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs" event={"ID":"9866232a-c748-4d81-a74f-1bc790f84bcc","Type":"ContainerDied","Data":"6bb8611e476eba6d54f991bfa6ab32701c94c7d125e3a51913b557e22a25edf5"}
Nov 02 13:59:13.171106 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:13.170781    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb8611e476eba6d54f991bfa6ab32701c94c7d125e3a51913b557e22a25edf5"
Nov 02 13:59:13.317581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:13.317526    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:13.317744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:13.317636    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:14.318612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.318575    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 13:59:14.319003 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.318645    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 13:59:14.360370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.360337    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"]
Nov 02 13:59:14.361134 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.361112    5572 cpu_manager.go:414] "containerMap: removing container" podUID="9866232a-c748-4d81-a74f-1bc790f84bcc" containerName="revoke-keys"
Nov 02 13:59:14.361134 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.361131    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9866232a-c748-4d81-a74f-1bc790f84bcc" containerName="revoke-keys"
Nov 02 13:59:14.361294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.361158    5572 cpu_manager.go:414] "containerMap: removing container" podUID="084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" containerName="revoke-keys"
Nov 02 13:59:14.361294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.361164    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" containerName="revoke-keys"
Nov 02 13:59:14.361294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.361261    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9866232a-c748-4d81-a74f-1bc790f84bcc" containerName="revoke-keys"
Nov 02 13:59:14.361294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.361273    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" containerName="revoke-keys"
Nov 02 13:59:14.365243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.365218    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.374866 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.374818    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:59:14.375189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.375170    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:14.375428 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.375414    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"]
Nov 02 13:59:14.387349 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.387308    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"]
Nov 02 13:59:14.387492 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.387424    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.400888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.399704    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.400888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.399799    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.400888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.399837    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf7th\" (UniqueName: \"kubernetes.io/projected/32943f10-f078-4c19-b12f-1eca8c092ff2-kube-api-access-vf7th\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.408606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.403226    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"]
Nov 02 13:59:14.501997 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.501957    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.502247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.502222    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w954r\" (UniqueName: \"kubernetes.io/projected/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-kube-api-access-w954r\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.502323 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.502278    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.502360 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.502347    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.502426 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.502409    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.502474 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.502453    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf7th\" (UniqueName: \"kubernetes.io/projected/32943f10-f078-4c19-b12f-1eca8c092ff2-kube-api-access-vf7th\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.503234 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.503207    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.505388 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.505362    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.515972 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.515942    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf7th\" (UniqueName: \"kubernetes.io/projected/32943f10-f078-4c19-b12f-1eca8c092ff2-kube-api-access-vf7th\") pod \"maas-api-revoke-sub-3372cd3d-5sm4d\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.604632 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.604515    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.604945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.604914    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w954r\" (UniqueName: \"kubernetes.io/projected/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-kube-api-access-w954r\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.605070 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.604973    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.605653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.605629    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.607231 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.607205    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.618154 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.618123    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w954r\" (UniqueName: \"kubernetes.io/projected/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-kube-api-access-w954r\") pod \"maas-api-revoke-sub-0349bcbd-h8thl\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.706622 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.706550    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:14.722974 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.722948    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:14.776036 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:14.775954    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" podUID="5a012951-d303-4275-b064-d61ff42b0699" containerName="payload-pre-processing" containerID="cri-o://fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f" gracePeriod=25
Nov 02 13:59:15.217757 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:15.217727    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32943f10_f078_4c19_b12f_1eca8c092ff2.slice/crio-fe51d2eb021db77c1a78eacbd89805665cde7de68921166941ae9ce52dc6c319 WatchSource:0}: Error finding container fe51d2eb021db77c1a78eacbd89805665cde7de68921166941ae9ce52dc6c319: Status 404 returned error can't find the container with id fe51d2eb021db77c1a78eacbd89805665cde7de68921166941ae9ce52dc6c319
Nov 02 13:59:15.219266 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.219238    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"]
Nov 02 13:59:15.254052 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.254022    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"]
Nov 02 13:59:15.256331 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:15.256290    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c06f530_d33a_4feb_a704_9ac3c38bf2c7.slice/crio-017b349a01fb042112bf15d772175017b81bd59f2c112531ace438787507bb2c WatchSource:0}: Error finding container 017b349a01fb042112bf15d772175017b81bd59f2c112531ace438787507bb2c: Status 404 returned error can't find the container with id 017b349a01fb042112bf15d772175017b81bd59f2c112531ace438787507bb2c
Nov 02 13:59:15.335099 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.335056    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:15.337801 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.337630    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:15.339032 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.339013    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 13:59:15.683039 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.682997    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"]
Nov 02 13:59:15.687879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.687858    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.703977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.703901    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"]
Nov 02 13:59:15.814354 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.814317    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7fd\" (UniqueName: \"kubernetes.io/projected/8d873a3f-39ba-4af5-9076-4b34712d8a39-kube-api-access-dq7fd\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.814581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.814367    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.814678 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.814652    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.919113 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.918515    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7fd\" (UniqueName: \"kubernetes.io/projected/8d873a3f-39ba-4af5-9076-4b34712d8a39-kube-api-access-dq7fd\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.919321 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.918592    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.919704 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.919677    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.923387 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.923351    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.946367 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.945449    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:15.954482 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:15.954450    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7fd\" (UniqueName: \"kubernetes.io/projected/8d873a3f-39ba-4af5-9076-4b34712d8a39-kube-api-access-dq7fd\") pod \"maas-api-revoke-sub-8eb94677-sd7g5\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:16.013498 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.013464    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:16.198682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.198574    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl" event={"ID":"7c06f530-d33a-4feb-a704-9ac3c38bf2c7","Type":"ContainerStarted","Data":"1b3f69be7f70e40dd6f628ce34591056b13c5fc8b3770a2243cd358aee50cce9"}
Nov 02 13:59:16.198682 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.198651    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl" event={"ID":"7c06f530-d33a-4feb-a704-9ac3c38bf2c7","Type":"ContainerStarted","Data":"017b349a01fb042112bf15d772175017b81bd59f2c112531ace438787507bb2c"}
Nov 02 13:59:16.201369 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.201344    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d" event={"ID":"32943f10-f078-4c19-b12f-1eca8c092ff2","Type":"ContainerStarted","Data":"a77094809484a0c67dfa97fd45618ad2468f80b75be08e3245652f64aa0035d5"}
Nov 02 13:59:16.201516 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.201386    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d" event={"ID":"32943f10-f078-4c19-b12f-1eca8c092ff2","Type":"ContainerStarted","Data":"fe51d2eb021db77c1a78eacbd89805665cde7de68921166941ae9ce52dc6c319"}
Nov 02 13:59:16.219097 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.219046    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl" podStartSLOduration=2.219028577 podStartE2EDuration="2.219028577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:16.218127613 +0000 UTC m=+932.796216501" watchObservedRunningTime="2026-11-02 13:59:16.219028577 +0000 UTC m=+932.797117447"
Nov 02 13:59:16.249432 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.249383    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d" podStartSLOduration=2.249368054 podStartE2EDuration="2.249368054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:16.246534689 +0000 UTC m=+932.824623557" watchObservedRunningTime="2026-11-02 13:59:16.249368054 +0000 UTC m=+932.827456920"
Nov 02 13:59:16.297967 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:16.297936    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d873a3f_39ba_4af5_9076_4b34712d8a39.slice/crio-a22466e2ece42b377284590d1dde6c8b6d56b8cc3a2bb880730908ab74bba62b WatchSource:0}: Error finding container a22466e2ece42b377284590d1dde6c8b6d56b8cc3a2bb880730908ab74bba62b: Status 404 returned error can't find the container with id a22466e2ece42b377284590d1dde6c8b6d56b8cc3a2bb880730908ab74bba62b
Nov 02 13:59:16.300297 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:16.300272    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"]
Nov 02 13:59:17.211297 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:17.211266    5572 generic.go:356] "Generic (PLEG): container finished" podID="7c06f530-d33a-4feb-a704-9ac3c38bf2c7" containerID="1b3f69be7f70e40dd6f628ce34591056b13c5fc8b3770a2243cd358aee50cce9" exitCode=0
Nov 02 13:59:17.211784 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:17.211358    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl" event={"ID":"7c06f530-d33a-4feb-a704-9ac3c38bf2c7","Type":"ContainerDied","Data":"1b3f69be7f70e40dd6f628ce34591056b13c5fc8b3770a2243cd358aee50cce9"}
Nov 02 13:59:17.213133 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:17.213110    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5" event={"ID":"8d873a3f-39ba-4af5-9076-4b34712d8a39","Type":"ContainerStarted","Data":"ac651213e83067a4c53f9ecd555f0604a5e69ce8494d7beb2f7cd0ca51c42506"}
Nov 02 13:59:17.213228 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:17.213153    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5" event={"ID":"8d873a3f-39ba-4af5-9076-4b34712d8a39","Type":"ContainerStarted","Data":"a22466e2ece42b377284590d1dde6c8b6d56b8cc3a2bb880730908ab74bba62b"}
Nov 02 13:59:17.214809 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:17.214764    5572 generic.go:356] "Generic (PLEG): container finished" podID="32943f10-f078-4c19-b12f-1eca8c092ff2" containerID="a77094809484a0c67dfa97fd45618ad2468f80b75be08e3245652f64aa0035d5" exitCode=0
Nov 02 13:59:17.214920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:17.214853    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d" event={"ID":"32943f10-f078-4c19-b12f-1eca8c092ff2","Type":"ContainerDied","Data":"a77094809484a0c67dfa97fd45618ad2468f80b75be08e3245652f64aa0035d5"}
Nov 02 13:59:18.226852 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:18.226808    5572 generic.go:356] "Generic (PLEG): container finished" podID="8d873a3f-39ba-4af5-9076-4b34712d8a39" containerID="ac651213e83067a4c53f9ecd555f0604a5e69ce8494d7beb2f7cd0ca51c42506" exitCode=0
Nov 02 13:59:18.227304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:18.226877    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5" event={"ID":"8d873a3f-39ba-4af5-9076-4b34712d8a39","Type":"ContainerDied","Data":"ac651213e83067a4c53f9ecd555f0604a5e69ce8494d7beb2f7cd0ca51c42506"}
Nov 02 13:59:19.299586 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.292772    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:19.459589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.452662    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:19.482594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.460058    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:59:19.522235 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.456272    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" containerID="cri-o://0aaa7e67c4abd5256d856e2d7af88e47e87bb0103b09af41ffc72f003a44c1a9" gracePeriod=45
Nov 02 13:59:19.537804 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.456127    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:59:19.538745 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.498425    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:19.627797 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.627747    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:19.698653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.653818    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:19.698653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.696419    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:19.813807 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.813759    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:19.814473 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.814441    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:59:19.817071 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.817045    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:59:19.892355 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.892310    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:59:19.916584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:19.904944    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.250840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.249179    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.250840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.250600    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 02 13:59:20.309625 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.305706    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" podUID="5a012951-d303-4275-b064-d61ff42b0699" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:59:20.332606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.310291    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:59:20.355589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.337689    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:59:20.358063 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.358028    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:59:20.517440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.500356    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.517440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.510715    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.552907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.527746    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.552907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.528289    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:59:20.552907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.548782    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.552907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.550434    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:59:20.552907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.551619    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:59:20.552907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.551940    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:59:20.649848 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.647096    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:59:20.698621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.650366    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:59:20.698621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.661118    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.698621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.661378    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:59:20.698621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.676527    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:20.700137 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.699471    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.703714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.703678    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:59:20.903083 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.896785    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.925668 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.906707    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.914748    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.949190    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.916074    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.916114    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.916135    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.950897    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.950944    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:59:20.993998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:20.950997    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:59:21.129591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.121972    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-kube-api-access-w954r\" (UniqueName: \"kubernetes.io/projected/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-kube-api-access-w954r\") pod \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") "
Nov 02 13:59:21.129591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.122081    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-xks-ca\") pod \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") "
Nov 02 13:59:21.129591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.122142    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-openshift-ca\") pod \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\" (UID: \"7c06f530-d33a-4feb-a704-9ac3c38bf2c7\") "
Nov 02 13:59:21.554935 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.373951    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:21.554935 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.503044    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:21.648587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.648283    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-openshift-ca" pod "7c06f530-d33a-4feb-a704-9ac3c38bf2c7" (UID: "7c06f530-d33a-4feb-a704-9ac3c38bf2c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:21.686631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.679057    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-kube-api-access-w954r" pod "7c06f530-d33a-4feb-a704-9ac3c38bf2c7" (UID: "7c06f530-d33a-4feb-a704-9ac3c38bf2c7"). InnerVolumeSpecName "kube-api-access-w954r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:21.819638 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.813309    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" containerID="cri-o://26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3" gracePeriod=43
Nov 02 13:59:21.955437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.830173    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:21.955437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.839519    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:59:21.955437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:21.840294    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:22.087902 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.035921    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:59:22.092593 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.090510    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-xks-ca" pod "7c06f530-d33a-4feb-a704-9ac3c38bf2c7" (UID: "7c06f530-d33a-4feb-a704-9ac3c38bf2c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:22.096590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.094284    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:59:22.096590 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.094353    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:22.336740 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.336690    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:22.405685 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.398532    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:22.405685 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.398801    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:22.430587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.423318    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused"
Nov 02 13:59:22.482923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.458714    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:22.505671 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.499317    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:22.541842 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.527216    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"
Nov 02 13:59:22.541842 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.527700    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:22.635592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.626631    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w954r\" (UniqueName: \"kubernetes.io/projected/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-kube-api-access-w954r\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:22.804224 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.667625    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.062508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.805237    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 02 13:59:23.062508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.844265    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.062508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.060446    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 02 13:59:23.062508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.890382    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.062508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.873201    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:59:23.116607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.891201    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.120594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.967209    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-xks-ca\") pod \"32943f10-f078-4c19-b12f-1eca8c092ff2\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") "
Nov 02 13:59:23.120594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.118061    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-openshift-ca\") pod \"32943f10-f078-4c19-b12f-1eca8c092ff2\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") "
Nov 02 13:59:23.120594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.118104    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32943f10-f078-4c19-b12f-1eca8c092ff2-kube-api-access-vf7th\" (UniqueName: \"kubernetes.io/projected/32943f10-f078-4c19-b12f-1eca8c092ff2-kube-api-access-vf7th\") pod \"32943f10-f078-4c19-b12f-1eca8c092ff2\" (UID: \"32943f10-f078-4c19-b12f-1eca8c092ff2\") "
Nov 02 13:59:23.136848 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:23.135416    5572 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.421s"
Nov 02 13:59:23.136848 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.135536    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:59:23.136848 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.135738    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl" event={"ID":"7c06f530-d33a-4feb-a704-9ac3c38bf2c7","Type":"ContainerDied","Data":"017b349a01fb042112bf15d772175017b81bd59f2c112531ace438787507bb2c"}
Nov 02 13:59:23.136848 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.135776    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="017b349a01fb042112bf15d772175017b81bd59f2c112531ace438787507bb2c"
Nov 02 13:59:23.174680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.144429    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c06f530-d33a-4feb-a704-9ac3c38bf2c7-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:23.174680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.030916    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:59:23.174680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:22.890634    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.174680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.030984    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:59:23.174680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.041787    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.178787 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.176472    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-xks-ca" pod "32943f10-f078-4c19-b12f-1eca8c092ff2" (UID: "32943f10-f078-4c19-b12f-1eca8c092ff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:23.305598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.299117    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:59:23.305598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.299186    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.305598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.301523    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:23.347457 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:23.347333    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:59:23.401594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.374810    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 02 13:59:23.518648 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.517601    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:59:23.563859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.563819    5572 patch_prober.go:29] interesting pod/thanos-querier-55c454df6-wgrrz container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.12:9091/-/healthy\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:59:23.564147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.564112    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-55c454df6-wgrrz" podUID="8cfd9f5a-4325-4769-9b66-8ef4dc715b5c" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/healthy\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.655647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.039917    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kbtw9" podUID="ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.33:50052\" within 1s: context deadline exceeded"
Nov 02 13:59:23.655647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.649865    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.742729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.742676    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.863588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.863109    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused"
Nov 02 13:59:23.878715 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.878669    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q595n" podUID="6b462672-55fe-42b8-b39f-52b0de99145a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 02 13:59:23.891588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.880679    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:59:23.916862 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.916771    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:23.952617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:23.936091    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:24.099054 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.099010    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-openshift-ca" pod "32943f10-f078-4c19-b12f-1eca8c092ff2" (UID: "32943f10-f078-4c19-b12f-1eca8c092ff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:24.135271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.135219    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:24.139943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.139902    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:59:24.191579 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.187155    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:59:24.214211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.214166    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:24.363065 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.229296    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:59:24.388269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.157862    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 13:59:24.388269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.229815    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:24.388269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.367700    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32943f10-f078-4c19-b12f-1eca8c092ff2-kube-api-access-vf7th" pod "32943f10-f078-4c19-b12f-1eca8c092ff2" (UID: "32943f10-f078-4c19-b12f-1eca8c092ff2"). InnerVolumeSpecName "kube-api-access-vf7th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:24.388269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.157903    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:59:24.404456 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.404413    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32943f10-f078-4c19-b12f-1eca8c092ff2-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:24.418666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.418627    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf7th\" (UniqueName: \"kubernetes.io/projected/32943f10-f078-4c19-b12f-1eca8c092ff2-kube-api-access-vf7th\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:24.419023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.418993    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 02 13:59:24.598653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.542599    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused"
Nov 02 13:59:24.598653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.585309    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:24.703124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.654644    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:59:24.941345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:24.941284    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:25.107582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.102978    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 02 13:59:25.120181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.107847    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:25.120508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.107880    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Nov 02 13:59:25.193140 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.193049    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:25.212582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.140198    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 13:59:25.239681 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.232854    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 02 13:59:25.255687 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.254300    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:25.262267 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:25.262234    5572 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.698s"
Nov 02 13:59:25.262454 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.262367    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"]
Nov 02 13:59:25.263404 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263379    5572 cpu_manager.go:414] "containerMap: removing container" podUID="8d873a3f-39ba-4af5-9076-4b34712d8a39" containerName="revoke-keys"
Nov 02 13:59:25.263534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263405    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d873a3f-39ba-4af5-9076-4b34712d8a39" containerName="revoke-keys"
Nov 02 13:59:25.263534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263464    5572 cpu_manager.go:414] "containerMap: removing container" podUID="32943f10-f078-4c19-b12f-1eca8c092ff2" containerName="revoke-keys"
Nov 02 13:59:25.263534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263474    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32943f10-f078-4c19-b12f-1eca8c092ff2" containerName="revoke-keys"
Nov 02 13:59:25.263534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263491    5572 cpu_manager.go:414] "containerMap: removing container" podUID="7c06f530-d33a-4feb-a704-9ac3c38bf2c7" containerName="revoke-keys"
Nov 02 13:59:25.263534 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263498    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c06f530-d33a-4feb-a704-9ac3c38bf2c7" containerName="revoke-keys"
Nov 02 13:59:25.263850 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263834    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="32943f10-f078-4c19-b12f-1eca8c092ff2" containerName="revoke-keys"
Nov 02 13:59:25.263949 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.263938    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c06f530-d33a-4feb-a704-9ac3c38bf2c7" containerName="revoke-keys"
Nov 02 13:59:25.264038 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.264028    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d873a3f-39ba-4af5-9076-4b34712d8a39" containerName="revoke-keys"
Nov 02 13:59:25.264491 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.264224    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"
Nov 02 13:59:25.301213 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.300765    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused"
Nov 02 13:59:25.306745 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.306646    5572 generic.go:356] "Generic (PLEG): container finished" podID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerID="0aaa7e67c4abd5256d856e2d7af88e47e87bb0103b09af41ffc72f003a44c1a9" exitCode=0
Nov 02 13:59:25.306943 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.306920    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:59:25.307014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.306966    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"]
Nov 02 13:59:25.307014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.306995    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.307097 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.307053    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:59:25.307176 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.307163    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:59:25.307220 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.307215    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:59:25.307258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.307235    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5" event={"ID":"8d873a3f-39ba-4af5-9076-4b34712d8a39","Type":"ContainerDied","Data":"a22466e2ece42b377284590d1dde6c8b6d56b8cc3a2bb880730908ab74bba62b"}
Nov 02 13:59:25.307293 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.307263    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22466e2ece42b377284590d1dde6c8b6d56b8cc3a2bb880730908ab74bba62b"
Nov 02 13:59:25.307328 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.307308    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:59:25.308265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308180    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:59:25.308265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308221    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d" event={"ID":"32943f10-f078-4c19-b12f-1eca8c092ff2","Type":"ContainerDied","Data":"fe51d2eb021db77c1a78eacbd89805665cde7de68921166941ae9ce52dc6c319"}
Nov 02 13:59:25.308375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308264    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe51d2eb021db77c1a78eacbd89805665cde7de68921166941ae9ce52dc6c319"
Nov 02 13:59:25.308375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308278    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" event={"ID":"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6","Type":"ContainerDied","Data":"0aaa7e67c4abd5256d856e2d7af88e47e87bb0103b09af41ffc72f003a44c1a9"}
Nov 02 13:59:25.308375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308318    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:59:25.308375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308355    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:59:25.309105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308395    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:59:25.309105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308718    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:59:25.309105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308764    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:59:25.309105 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.308831    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:59:25.341915 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:25.341870    5572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd70e1a5e_b2a0_4a0f_a0f8_693697f0dce6.slice/crio-0aaa7e67c4abd5256d856e2d7af88e47e87bb0103b09af41ffc72f003a44c1a9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd70e1a5e_b2a0_4a0f_a0f8_693697f0dce6.slice/crio-conmon-0aaa7e67c4abd5256d856e2d7af88e47e87bb0103b09af41ffc72f003a44c1a9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c246434_da6e_4c64_8743_b3778b117e7a.slice/crio-26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:59:25.423070 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.423042    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-openshift-ca\") pod \"8d873a3f-39ba-4af5-9076-4b34712d8a39\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") "
Nov 02 13:59:25.423247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.423148    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-xks-ca\") pod \"8d873a3f-39ba-4af5-9076-4b34712d8a39\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") "
Nov 02 13:59:25.423247 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.423220    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d873a3f-39ba-4af5-9076-4b34712d8a39-kube-api-access-dq7fd\" (UniqueName: \"kubernetes.io/projected/8d873a3f-39ba-4af5-9076-4b34712d8a39-kube-api-access-dq7fd\") pod \"8d873a3f-39ba-4af5-9076-4b34712d8a39\" (UID: \"8d873a3f-39ba-4af5-9076-4b34712d8a39\") "
Nov 02 13:59:25.423481 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.423454    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-openshift-ca" pod "8d873a3f-39ba-4af5-9076-4b34712d8a39" (UID: "8d873a3f-39ba-4af5-9076-4b34712d8a39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:25.423765 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.423738    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8h9v\" (UniqueName: \"kubernetes.io/projected/7bd1839d-ea2a-4472-89e1-8e414e743dad-kube-api-access-n8h9v\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.423923 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.423903    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.423999 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.423968    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.426181 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.426155    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.426352 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.426331    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-xks-ca" pod "8d873a3f-39ba-4af5-9076-4b34712d8a39" (UID: "8d873a3f-39ba-4af5-9076-4b34712d8a39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:25.426430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.426364    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d873a3f-39ba-4af5-9076-4b34712d8a39-kube-api-access-dq7fd" pod "8d873a3f-39ba-4af5-9076-4b34712d8a39" (UID: "8d873a3f-39ba-4af5-9076-4b34712d8a39"). InnerVolumeSpecName "kube-api-access-dq7fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:25.453425 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.453358    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:59:25.527764 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.527726    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8h9v\" (UniqueName: \"kubernetes.io/projected/7bd1839d-ea2a-4472-89e1-8e414e743dad-kube-api-access-n8h9v\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.527967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.527835    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.527967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.527872    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.527967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.527947    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d873a3f-39ba-4af5-9076-4b34712d8a39-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.527967 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.527963    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7fd\" (UniqueName: \"kubernetes.io/projected/8d873a3f-39ba-4af5-9076-4b34712d8a39-kube-api-access-dq7fd\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.528821 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.528674    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.533178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.533154    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.543058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.543032    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8h9v\" (UniqueName: \"kubernetes.io/projected/7bd1839d-ea2a-4472-89e1-8e414e743dad-kube-api-access-n8h9v\") pod \"maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.628938 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.628897    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-kube-api-access-hz88b\" (UniqueName: \"kubernetes.io/projected/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-kube-api-access-hz88b\") pod \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") "
Nov 02 13:59:25.629130 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.629081    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-dshm\") pod \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") "
Nov 02 13:59:25.629194 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.629175    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-home\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-home\") pod \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") "
Nov 02 13:59:25.629268 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.629254    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tmp-dir\") pod \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") "
Nov 02 13:59:25.629382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.629362    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-model-cache\") pod \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") "
Nov 02 13:59:25.629588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.629454    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-home" pod "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" (UID: "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:25.629588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.629468    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tls-certs\") pod \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\" (UID: \"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6\") "
Nov 02 13:59:25.629745 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.629593    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tmp-dir" pod "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" (UID: "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:25.630156 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.630126    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-model-cache" pod "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" (UID: "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:25.631073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.631051    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.631200 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.631082    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.631200 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.631099    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.631838 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.631811    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-dshm" pod "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" (UID: "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:25.631838 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.631814    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-kube-api-access-hz88b" pod "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" (UID: "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6"). InnerVolumeSpecName "kube-api-access-hz88b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:25.632696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.632674    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tls-certs" pod "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" (UID: "d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:25.650994 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.650959    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:25.675989 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.675956    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"]
Nov 02 13:59:25.686998 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.686964    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"]
Nov 02 13:59:25.699826 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.699794    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"]
Nov 02 13:59:25.733196 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.732674    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.733196 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.732698    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz88b\" (UniqueName: \"kubernetes.io/projected/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-kube-api-access-hz88b\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.733196 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.732708    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.965263 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:25.964965    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"]
Nov 02 13:59:26.204940 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.204916    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:59:26.318532 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.318474    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg" event={"ID":"7bd1839d-ea2a-4472-89e1-8e414e743dad","Type":"ContainerStarted","Data":"95b481ef3af589179edd7e03cda6d9902c581489e2dbafc47dde43d9fabca1cb"}
Nov 02 13:59:26.320692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.320657    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5" event={"ID":"d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6","Type":"ContainerDied","Data":"761ae95c5fae1e1efd8d2dbee522bdb219d2909e8de6996a427a3478aa56cd79"}
Nov 02 13:59:26.320872 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.320720    5572 scope.go:122] "RemoveContainer" containerID="0aaa7e67c4abd5256d856e2d7af88e47e87bb0103b09af41ffc72f003a44c1a9"
Nov 02 13:59:26.320872 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.320669    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"
Nov 02 13:59:26.322767 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.322739    5572 generic.go:356] "Generic (PLEG): container finished" podID="5c246434-da6e-4c64-8743-b3778b117e7a" containerID="26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3" exitCode=0
Nov 02 13:59:26.323053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.322833    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"
Nov 02 13:59:26.323053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.322872    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" event={"ID":"5c246434-da6e-4c64-8743-b3778b117e7a","Type":"ContainerDied","Data":"26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3"}
Nov 02 13:59:26.323053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.322923    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs" event={"ID":"5c246434-da6e-4c64-8743-b3778b117e7a","Type":"ContainerDied","Data":"628de5c73493baff479eef4cd589cbbcaee9a41337efab6e03853abcc46227a7"}
Nov 02 13:59:26.323240 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.323102    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"
Nov 02 13:59:26.343122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.343090    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-model-cache\") pod \"5c246434-da6e-4c64-8743-b3778b117e7a\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") "
Nov 02 13:59:26.343417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.343387    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-model-cache" pod "5c246434-da6e-4c64-8743-b3778b117e7a" (UID: "5c246434-da6e-4c64-8743-b3778b117e7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:26.343653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.343631    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-dshm\") pod \"5c246434-da6e-4c64-8743-b3778b117e7a\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") "
Nov 02 13:59:26.343858 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.343842    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-home\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-home\") pod \"5c246434-da6e-4c64-8743-b3778b117e7a\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") "
Nov 02 13:59:26.344149 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.344066    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c246434-da6e-4c64-8743-b3778b117e7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c246434-da6e-4c64-8743-b3778b117e7a-tls-certs\") pod \"5c246434-da6e-4c64-8743-b3778b117e7a\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") "
Nov 02 13:59:26.344149 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.344128    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-tmp-dir\") pod \"5c246434-da6e-4c64-8743-b3778b117e7a\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") "
Nov 02 13:59:26.344621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.344173    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c246434-da6e-4c64-8743-b3778b117e7a-kube-api-access-65gwp\" (UniqueName: \"kubernetes.io/projected/5c246434-da6e-4c64-8743-b3778b117e7a-kube-api-access-65gwp\") pod \"5c246434-da6e-4c64-8743-b3778b117e7a\" (UID: \"5c246434-da6e-4c64-8743-b3778b117e7a\") "
Nov 02 13:59:26.344621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.344329    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-home" pod "5c246434-da6e-4c64-8743-b3778b117e7a" (UID: "5c246434-da6e-4c64-8743-b3778b117e7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:26.344881 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.344856    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-tmp-dir" pod "5c246434-da6e-4c64-8743-b3778b117e7a" (UID: "5c246434-da6e-4c64-8743-b3778b117e7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:26.345127 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.345105    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:26.345213 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.345134    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:26.345279 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.345268    5572 scope.go:122] "RemoveContainer" containerID="26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3"
Nov 02 13:59:26.349681 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.349655    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-dshm" pod "5c246434-da6e-4c64-8743-b3778b117e7a" (UID: "5c246434-da6e-4c64-8743-b3778b117e7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:26.349833 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.349804    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c246434-da6e-4c64-8743-b3778b117e7a-tls-certs" pod "5c246434-da6e-4c64-8743-b3778b117e7a" (UID: "5c246434-da6e-4c64-8743-b3778b117e7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:26.350636 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.350548    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c246434-da6e-4c64-8743-b3778b117e7a-kube-api-access-65gwp" pod "5c246434-da6e-4c64-8743-b3778b117e7a" (UID: "5c246434-da6e-4c64-8743-b3778b117e7a"). InnerVolumeSpecName "kube-api-access-65gwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:26.363388 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.363361    5572 scope.go:122] "RemoveContainer" containerID="26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3"
Nov 02 13:59:26.363837 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:26.363807    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3\": container with ID starting with 26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3 not found: ID does not exist" containerID="26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3"
Nov 02 13:59:26.363945 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.363850    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3"} err="failed to get container status \"26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3\": rpc error: code = NotFound desc = could not find container \"26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3\": container with ID starting with 26d79745c932e238ed89cb258436692f4612ad88957c5c49a298c07b8bf196d3 not found: ID does not exist"
Nov 02 13:59:26.449979 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.449117    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c246434-da6e-4c64-8743-b3778b117e7a-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:26.449979 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.449179    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:26.449979 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.449195    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65gwp\" (UniqueName: \"kubernetes.io/projected/5c246434-da6e-4c64-8743-b3778b117e7a-kube-api-access-65gwp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:26.449979 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.449212    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c246434-da6e-4c64-8743-b3778b117e7a-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:26.476888 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.476812    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"]
Nov 02 13:59:26.495584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.491500    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/simulator-w5-1d18fb-kserve-7885c684c6-r6sq5"]
Nov 02 13:59:26.650836 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.650808    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"]
Nov 02 13:59:26.660920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:26.660892    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1d18fb/distinct-w5-1d18fb-kserve-6857c74968-989cs"]
Nov 02 13:59:27.334667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.334628    5572 generic.go:356] "Generic (PLEG): container finished" podID="7bd1839d-ea2a-4472-89e1-8e414e743dad" containerID="615df1e5f07c2f7742c3f3651569d45e1b3f47bf2a8eb8fb444a9bedde0baacd" exitCode=0
Nov 02 13:59:27.335139 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.334710    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg" event={"ID":"7bd1839d-ea2a-4472-89e1-8e414e743dad","Type":"ContainerDied","Data":"615df1e5f07c2f7742c3f3651569d45e1b3f47bf2a8eb8fb444a9bedde0baacd"}
Nov 02 13:59:27.544876 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.544793    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" path="/var/lib/kubelet/pods/5c246434-da6e-4c64-8743-b3778b117e7a/volumes"
Nov 02 13:59:27.545333 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.545319    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" path="/var/lib/kubelet/pods/d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6/volumes"
Nov 02 13:59:27.987265 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.987233    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"]
Nov 02 13:59:27.987978 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.987964    5572 cpu_manager.go:414] "containerMap: removing container" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main"
Nov 02 13:59:27.988042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.987980    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main"
Nov 02 13:59:27.988042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.987993    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main"
Nov 02 13:59:27.988042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.987997    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main"
Nov 02 13:59:27.988132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.988097    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c246434-da6e-4c64-8743-b3778b117e7a" containerName="main"
Nov 02 13:59:27.988132 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.988106    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70e1a5e-b2a0-4a0f-a0f8-693697f0dce6" containerName="main"
Nov 02 13:59:27.991883 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:27.991867    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.003634 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.003608    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"]
Nov 02 13:59:28.076234 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.076197    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.076433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.076388    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.076595 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.076547    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hvl\" (UniqueName: \"kubernetes.io/projected/5f3aee62-baf3-4d58-82f0-33e5da695717-kube-api-access-j5hvl\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.178060 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.178030    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hvl\" (UniqueName: \"kubernetes.io/projected/5f3aee62-baf3-4d58-82f0-33e5da695717-kube-api-access-j5hvl\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.178255 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.178132    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.178318 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.178253    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.179002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.178978    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.222587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.204279    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.222587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.209968    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hvl\" (UniqueName: \"kubernetes.io/projected/5f3aee62-baf3-4d58-82f0-33e5da695717-kube-api-access-j5hvl\") pod \"maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.327582 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.324872    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:28.566879 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.566854    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:28.628987 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:28.628955    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3aee62_baf3_4d58_82f0_33e5da695717.slice/crio-4ef7d4ddf029e2a8fd6cdf37e92cdebca51112c8edfb7044db21e5bef75d70a7 WatchSource:0}: Error finding container 4ef7d4ddf029e2a8fd6cdf37e92cdebca51112c8edfb7044db21e5bef75d70a7: Status 404 returned error can't find the container with id 4ef7d4ddf029e2a8fd6cdf37e92cdebca51112c8edfb7044db21e5bef75d70a7
Nov 02 13:59:28.631941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.631917    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"]
Nov 02 13:59:28.693969 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.693941    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd1839d-ea2a-4472-89e1-8e414e743dad-kube-api-access-n8h9v\" (UniqueName: \"kubernetes.io/projected/7bd1839d-ea2a-4472-89e1-8e414e743dad-kube-api-access-n8h9v\") pod \"7bd1839d-ea2a-4472-89e1-8e414e743dad\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") "
Nov 02 13:59:28.694131 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.693989    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-openshift-ca\") pod \"7bd1839d-ea2a-4472-89e1-8e414e743dad\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") "
Nov 02 13:59:28.694131 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.694098    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-xks-ca\") pod \"7bd1839d-ea2a-4472-89e1-8e414e743dad\" (UID: \"7bd1839d-ea2a-4472-89e1-8e414e743dad\") "
Nov 02 13:59:28.694382 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.694357    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-openshift-ca" pod "7bd1839d-ea2a-4472-89e1-8e414e743dad" (UID: "7bd1839d-ea2a-4472-89e1-8e414e743dad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:28.694840 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.694816    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:28.696380 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.696354    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-xks-ca" pod "7bd1839d-ea2a-4472-89e1-8e414e743dad" (UID: "7bd1839d-ea2a-4472-89e1-8e414e743dad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:28.696380 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.696361    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd1839d-ea2a-4472-89e1-8e414e743dad-kube-api-access-n8h9v" pod "7bd1839d-ea2a-4472-89e1-8e414e743dad" (UID: "7bd1839d-ea2a-4472-89e1-8e414e743dad"). InnerVolumeSpecName "kube-api-access-n8h9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:28.796708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.796677    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8h9v\" (UniqueName: \"kubernetes.io/projected/7bd1839d-ea2a-4472-89e1-8e414e743dad-kube-api-access-n8h9v\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:28.796708 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:28.796706    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd1839d-ea2a-4472-89e1-8e414e743dad-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:29.421135 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:29.421089    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg" event={"ID":"7bd1839d-ea2a-4472-89e1-8e414e743dad","Type":"ContainerDied","Data":"95b481ef3af589179edd7e03cda6d9902c581489e2dbafc47dde43d9fabca1cb"}
Nov 02 13:59:29.421325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:29.421151    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b481ef3af589179edd7e03cda6d9902c581489e2dbafc47dde43d9fabca1cb"
Nov 02 13:59:29.421325 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:29.421104    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"
Nov 02 13:59:29.424034 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:29.424009    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt" event={"ID":"5f3aee62-baf3-4d58-82f0-33e5da695717","Type":"ContainerStarted","Data":"5c65d96b7eabe9f7e4916d4d3d7f005eb73ced6249fac592c161ad8288c39cd4"}
Nov 02 13:59:29.424143 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:29.424048    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt" event={"ID":"5f3aee62-baf3-4d58-82f0-33e5da695717","Type":"ContainerStarted","Data":"4ef7d4ddf029e2a8fd6cdf37e92cdebca51112c8edfb7044db21e5bef75d70a7"}
Nov 02 13:59:29.445798 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:29.445743    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt" podStartSLOduration=2.44572881 podStartE2EDuration="2.44572881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:29.442007259 +0000 UTC m=+946.020096126" watchObservedRunningTime="2026-11-02 13:59:29.44572881 +0000 UTC m=+946.023817677"
Nov 02 13:59:30.056202 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:30.056159    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" podUID="5a012951-d303-4275-b064-d61ff42b0699" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:59:30.434028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:30.433988    5572 generic.go:356] "Generic (PLEG): container finished" podID="5f3aee62-baf3-4d58-82f0-33e5da695717" containerID="5c65d96b7eabe9f7e4916d4d3d7f005eb73ced6249fac592c161ad8288c39cd4" exitCode=0
Nov 02 13:59:30.434208 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:30.434079    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt" event={"ID":"5f3aee62-baf3-4d58-82f0-33e5da695717","Type":"ContainerDied","Data":"5c65d96b7eabe9f7e4916d4d3d7f005eb73ced6249fac592c161ad8288c39cd4"}
Nov 02 13:59:31.614238 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.614195    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:31.647592 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.647527    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-xks-ca\") pod \"5f3aee62-baf3-4d58-82f0-33e5da695717\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") "
Nov 02 13:59:31.647767 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.647625    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3aee62-baf3-4d58-82f0-33e5da695717-kube-api-access-j5hvl\" (UniqueName: \"kubernetes.io/projected/5f3aee62-baf3-4d58-82f0-33e5da695717-kube-api-access-j5hvl\") pod \"5f3aee62-baf3-4d58-82f0-33e5da695717\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") "
Nov 02 13:59:31.647896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.647873    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-openshift-ca\") pod \"5f3aee62-baf3-4d58-82f0-33e5da695717\" (UID: \"5f3aee62-baf3-4d58-82f0-33e5da695717\") "
Nov 02 13:59:31.648406 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.648375    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-openshift-ca" pod "5f3aee62-baf3-4d58-82f0-33e5da695717" (UID: "5f3aee62-baf3-4d58-82f0-33e5da695717"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:31.648742 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.648714    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:31.650022 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.649997    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-xks-ca" pod "5f3aee62-baf3-4d58-82f0-33e5da695717" (UID: "5f3aee62-baf3-4d58-82f0-33e5da695717"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:31.650716 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.650690    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3aee62-baf3-4d58-82f0-33e5da695717-kube-api-access-j5hvl" pod "5f3aee62-baf3-4d58-82f0-33e5da695717" (UID: "5f3aee62-baf3-4d58-82f0-33e5da695717"). InnerVolumeSpecName "kube-api-access-j5hvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:31.749819 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.749734    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f3aee62-baf3-4d58-82f0-33e5da695717-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:31.749819 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:31.749767    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5hvl\" (UniqueName: \"kubernetes.io/projected/5f3aee62-baf3-4d58-82f0-33e5da695717-kube-api-access-j5hvl\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:32.406753 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.406704    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"]
Nov 02 13:59:32.407440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.407426    5572 cpu_manager.go:414] "containerMap: removing container" podUID="7bd1839d-ea2a-4472-89e1-8e414e743dad" containerName="revoke-keys"
Nov 02 13:59:32.407483 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.407441    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd1839d-ea2a-4472-89e1-8e414e743dad" containerName="revoke-keys"
Nov 02 13:59:32.407483 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.407460    5572 cpu_manager.go:414] "containerMap: removing container" podUID="5f3aee62-baf3-4d58-82f0-33e5da695717" containerName="revoke-keys"
Nov 02 13:59:32.407483 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.407466    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3aee62-baf3-4d58-82f0-33e5da695717" containerName="revoke-keys"
Nov 02 13:59:32.407633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.407599    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd1839d-ea2a-4472-89e1-8e414e743dad" containerName="revoke-keys"
Nov 02 13:59:32.407633 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.407612    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3aee62-baf3-4d58-82f0-33e5da695717" containerName="revoke-keys"
Nov 02 13:59:32.411803 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.411787    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.421341 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.421194    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"]
Nov 02 13:59:32.452502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.452468    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt" event={"ID":"5f3aee62-baf3-4d58-82f0-33e5da695717","Type":"ContainerDied","Data":"4ef7d4ddf029e2a8fd6cdf37e92cdebca51112c8edfb7044db21e5bef75d70a7"}
Nov 02 13:59:32.452688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.452513    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef7d4ddf029e2a8fd6cdf37e92cdebca51112c8edfb7044db21e5bef75d70a7"
Nov 02 13:59:32.452688 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.452603    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"
Nov 02 13:59:32.467190 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.467162    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.467341 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.467210    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.467502 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.467479    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fhn\" (UniqueName: \"kubernetes.io/projected/bbdbc41f-73f1-4093-8340-d0b621786ae6-kube-api-access-g7fhn\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.568844 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.568805    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.569014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.568856    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.569014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.568965    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fhn\" (UniqueName: \"kubernetes.io/projected/bbdbc41f-73f1-4093-8340-d0b621786ae6-kube-api-access-g7fhn\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.569744 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.569722    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.571530 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.571509    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.581116 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.581093    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fhn\" (UniqueName: \"kubernetes.io/projected/bbdbc41f-73f1-4093-8340-d0b621786ae6-kube-api-access-g7fhn\") pod \"maas-api-revoke-sub-06eba304-6s45v\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:32.645187 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.645150    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"]
Nov 02 13:59:32.676649 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.676410    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"]
Nov 02 13:59:32.729397 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:32.729361    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:33.142584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:33.122928    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"]
Nov 02 13:59:33.516596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:33.516297    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v" event={"ID":"bbdbc41f-73f1-4093-8340-d0b621786ae6","Type":"ContainerStarted","Data":"e47afaf0befeabf74bd9ba16a3c28491441190aec14c4cd056b2c62b73d7fbee"}
Nov 02 13:59:34.530887 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.530840    5572 generic.go:356] "Generic (PLEG): container finished" podID="bbdbc41f-73f1-4093-8340-d0b621786ae6" containerID="e3340f0d06c6f6978cd4b7a0bd6ffd2be3bcd5b4b3354d67597495cdac66e04d" exitCode=0
Nov 02 13:59:34.531406 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.530987    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v" event={"ID":"bbdbc41f-73f1-4093-8340-d0b621786ae6","Type":"ContainerDied","Data":"e3340f0d06c6f6978cd4b7a0bd6ffd2be3bcd5b4b3354d67597495cdac66e04d"}
Nov 02 13:59:34.890001 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.889972    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"]
Nov 02 13:59:34.895373 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.895338    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:34.910242 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.909010    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"]
Nov 02 13:59:34.957658 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.957541    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:34.957924 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.957893    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:34.958075 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:34.958057    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl5m7\" (UniqueName: \"kubernetes.io/projected/996429a6-9cfc-4a67-9905-02c678287bb5-kube-api-access-zl5m7\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.060122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.060089    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.060322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.060214    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.060322 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.060290    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl5m7\" (UniqueName: \"kubernetes.io/projected/996429a6-9cfc-4a67-9905-02c678287bb5-kube-api-access-zl5m7\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.060992 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.060966    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.063384 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.063357    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.079327 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.079301    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl5m7\" (UniqueName: \"kubernetes.io/projected/996429a6-9cfc-4a67-9905-02c678287bb5-kube-api-access-zl5m7\") pod \"maas-api-revoke-sub-c658ee07-4txhw\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.115684 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.115655    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:59:35.161776 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.161746    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a012951-d303-4275-b064-d61ff42b0699-kube-api-access-6czht\" (UniqueName: \"kubernetes.io/projected/5a012951-d303-4275-b064-d61ff42b0699-kube-api-access-6czht\") pod \"5a012951-d303-4275-b064-d61ff42b0699\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") "
Nov 02 13:59:35.162007 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.161987    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a012951-d303-4275-b064-d61ff42b0699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a012951-d303-4275-b064-d61ff42b0699-plugins-config-volume\") pod \"5a012951-d303-4275-b064-d61ff42b0699\" (UID: \"5a012951-d303-4275-b064-d61ff42b0699\") "
Nov 02 13:59:35.162672 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.162456    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a012951-d303-4275-b064-d61ff42b0699-plugins-config-volume" pod "5a012951-d303-4275-b064-d61ff42b0699" (UID: "5a012951-d303-4275-b064-d61ff42b0699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:35.162824 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.162751    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a012951-d303-4275-b064-d61ff42b0699-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:35.164437 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.164388    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a012951-d303-4275-b064-d61ff42b0699-kube-api-access-6czht" pod "5a012951-d303-4275-b064-d61ff42b0699" (UID: "5a012951-d303-4275-b064-d61ff42b0699"). InnerVolumeSpecName "kube-api-access-6czht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:35.224452 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.224420    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:35.264624 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.264596    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czht\" (UniqueName: \"kubernetes.io/projected/5a012951-d303-4275-b064-d61ff42b0699-kube-api-access-6czht\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:35.536358 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:35.536322    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996429a6_9cfc_4a67_9905_02c678287bb5.slice/crio-67e9a2ddc4006dbd3b00233c02ada7c1f52e2eddc60331163fde8764a08e86af WatchSource:0}: Error finding container 67e9a2ddc4006dbd3b00233c02ada7c1f52e2eddc60331163fde8764a08e86af: Status 404 returned error can't find the container with id 67e9a2ddc4006dbd3b00233c02ada7c1f52e2eddc60331163fde8764a08e86af
Nov 02 13:59:35.543802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.543776    5572 generic.go:356] "Generic (PLEG): container finished" podID="5a012951-d303-4275-b064-d61ff42b0699" containerID="fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f" exitCode=0
Nov 02 13:59:35.543920 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.543893    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"
Nov 02 13:59:35.545316 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.545280    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" event={"ID":"5a012951-d303-4275-b064-d61ff42b0699","Type":"ContainerDied","Data":"fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f"}
Nov 02 13:59:35.545405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.545331    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84" event={"ID":"5a012951-d303-4275-b064-d61ff42b0699","Type":"ContainerDied","Data":"9579de7f7cf9f299931c85cc8812e27ac768b448dc5f7ee107aa7745e032ecea"}
Nov 02 13:59:35.545405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.545360    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"]
Nov 02 13:59:35.545405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.545398    5572 scope.go:122] "RemoveContainer" containerID="fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f"
Nov 02 13:59:35.592295 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.592256    5572 scope.go:122] "RemoveContainer" containerID="fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f"
Nov 02 13:59:35.592699 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:35.592672    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f\": container with ID starting with fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f not found: ID does not exist" containerID="fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f"
Nov 02 13:59:35.592808 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.592705    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f"} err="failed to get container status \"fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f\": rpc error: code = NotFound desc = could not find container \"fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f\": container with ID starting with fc3f4773a10e4a5a6caf4159ddbe84bdf76ddc25dad5417407881f85b6612c5f not found: ID does not exist"
Nov 02 13:59:35.595651 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.595627    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"]
Nov 02 13:59:35.606398 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.606375    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-1d18fb-766fdb9695-q8x84"]
Nov 02 13:59:35.723368 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.723344    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:35.772646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.772610    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-openshift-ca\") pod \"bbdbc41f-73f1-4093-8340-d0b621786ae6\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") "
Nov 02 13:59:35.772829 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.772737    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-xks-ca\") pod \"bbdbc41f-73f1-4093-8340-d0b621786ae6\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") "
Nov 02 13:59:35.772896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.772827    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdbc41f-73f1-4093-8340-d0b621786ae6-kube-api-access-g7fhn\" (UniqueName: \"kubernetes.io/projected/bbdbc41f-73f1-4093-8340-d0b621786ae6-kube-api-access-g7fhn\") pod \"bbdbc41f-73f1-4093-8340-d0b621786ae6\" (UID: \"bbdbc41f-73f1-4093-8340-d0b621786ae6\") "
Nov 02 13:59:35.772974 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.772953    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-openshift-ca" pod "bbdbc41f-73f1-4093-8340-d0b621786ae6" (UID: "bbdbc41f-73f1-4093-8340-d0b621786ae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:35.773644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.773619    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:35.775315 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.775293    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-xks-ca" pod "bbdbc41f-73f1-4093-8340-d0b621786ae6" (UID: "bbdbc41f-73f1-4093-8340-d0b621786ae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:35.775689 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.775665    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdbc41f-73f1-4093-8340-d0b621786ae6-kube-api-access-g7fhn" pod "bbdbc41f-73f1-4093-8340-d0b621786ae6" (UID: "bbdbc41f-73f1-4093-8340-d0b621786ae6"). InnerVolumeSpecName "kube-api-access-g7fhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:35.918424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.917897    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbdbc41f-73f1-4093-8340-d0b621786ae6-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:35.918424 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:35.917931    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7fhn\" (UniqueName: \"kubernetes.io/projected/bbdbc41f-73f1-4093-8340-d0b621786ae6-kube-api-access-g7fhn\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:36.413533 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.413497    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"]
Nov 02 13:59:36.564780 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.564753    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"
Nov 02 13:59:36.575607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.575512    5572 cpu_manager.go:414] "containerMap: removing container" podUID="5a012951-d303-4275-b064-d61ff42b0699" containerName="payload-pre-processing"
Nov 02 13:59:36.575607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.575549    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a012951-d303-4275-b064-d61ff42b0699" containerName="payload-pre-processing"
Nov 02 13:59:36.575946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.575649    5572 cpu_manager.go:414] "containerMap: removing container" podUID="bbdbc41f-73f1-4093-8340-d0b621786ae6" containerName="revoke-keys"
Nov 02 13:59:36.575946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.575664    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdbc41f-73f1-4093-8340-d0b621786ae6" containerName="revoke-keys"
Nov 02 13:59:36.575946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.575914    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdbc41f-73f1-4093-8340-d0b621786ae6" containerName="revoke-keys"
Nov 02 13:59:36.575946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.575935    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a012951-d303-4275-b064-d61ff42b0699" containerName="payload-pre-processing"
Nov 02 13:59:36.866618 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.865305    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw" podStartSLOduration=2.8652840619999997 podStartE2EDuration="2.865284062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:36.863429843 +0000 UTC m=+953.441518711" watchObservedRunningTime="2026-11-02 13:59:36.865284062 +0000 UTC m=+953.443372929"
Nov 02 13:59:36.923940 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.923887    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:59:37.000588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:36.958835    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:37.183066 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.182945    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw" event={"ID":"996429a6-9cfc-4a67-9905-02c678287bb5","Type":"ContainerStarted","Data":"28d9734a93eefe4f1c76fbe6d861026337b09e44f3902f14b81c8d7e69c629e5"}
Nov 02 13:59:37.183367 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.183340    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw" event={"ID":"996429a6-9cfc-4a67-9905-02c678287bb5","Type":"ContainerStarted","Data":"67e9a2ddc4006dbd3b00233c02ada7c1f52e2eddc60331163fde8764a08e86af"}
Nov 02 13:59:37.185205 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.185176    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"]
Nov 02 13:59:37.185669 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.185551    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v" event={"ID":"bbdbc41f-73f1-4093-8340-d0b621786ae6","Type":"ContainerDied","Data":"e47afaf0befeabf74bd9ba16a3c28491441190aec14c4cd056b2c62b73d7fbee"}
Nov 02 13:59:37.185807 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.185790    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e47afaf0befeabf74bd9ba16a3c28491441190aec14c4cd056b2c62b73d7fbee"
Nov 02 13:59:37.189617 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.189596    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.423829    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.423958    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.424028    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.424089    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.424286    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.424328    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jzz\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-kube-api-access-r4jzz\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.424354    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.424407    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.432656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.424468    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.544930    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545060    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545143    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545190    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545248    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545420    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545460    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jzz\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-kube-api-access-r4jzz\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545489    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.545543    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.546079    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-certs\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.546730    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-credential-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.547433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.547044    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-data\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.548581 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.548535    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-socket\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.549451 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.549429    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istiod-ca-cert\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.579124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.562067    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-token\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.579124 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.569392    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-podinfo\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.634791 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.620740    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-envoy\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.692548 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.692508    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a012951-d303-4275-b064-d61ff42b0699" path="/var/lib/kubelet/pods/5a012951-d303-4275-b064-d61ff42b0699/volumes"
Nov 02 13:59:37.722935 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.722899    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw" event={"ID":"996429a6-9cfc-4a67-9905-02c678287bb5","Type":"ContainerDied","Data":"28d9734a93eefe4f1c76fbe6d861026337b09e44f3902f14b81c8d7e69c629e5"}
Nov 02 13:59:37.723191 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.695137    5572 generic.go:356] "Generic (PLEG): container finished" podID="996429a6-9cfc-4a67-9905-02c678287bb5" containerID="28d9734a93eefe4f1c76fbe6d861026337b09e44f3902f14b81c8d7e69c629e5" exitCode=0
Nov 02 13:59:37.736642 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.733719    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jzz\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-kube-api-access-r4jzz\") pod \"e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") " pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:37.871598 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.868201    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:37.892606 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.889715    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:37.935597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.932228    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:59:37.935597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.934150    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:59:37.977591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:37.976875    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:38.134656 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.132804    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:38.138588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.136668    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:38.191597 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.189105    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:38.249408 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.248716    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:38.274616 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.250014    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:59:38.501786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.501682    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:59:38.503291 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.503261    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 13:59:38.505177 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.505152    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 13:59:38.516021 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.514615    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 02 13:59:38.887011 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.886965    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 13:59:38.945591 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.934504    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.014831 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.948032    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 13:59:39.014831 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:38.952962    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:59:39.199644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.148678    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.199644 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.149180    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:59:39.207585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.204685    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.207585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.205824    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:59:39.211635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.208325    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.211635 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.208919    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:59:39.278587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.269675    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.278587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.270783    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:59:39.278587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.271324    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:59:39.278587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.271380    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.629904    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.632116    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.632456    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.632743    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.633033    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.633090    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.633152    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:59:39.648585 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.633190    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:59:39.952691 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.907474    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.953654 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.949976    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:39.954342 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:39.950423    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 02 13:59:40.133243 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.133209    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:59:40.133683 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.133659    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"]
Nov 02 13:59:40.151023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.150901    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2" podUID="87666c0f-6310-470a-8242-b4345b50acbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:40.206795 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.206698    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:40.225239 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.225187    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded"
Nov 02 13:59:40.321796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.298345    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Nov 02 13:59:40.321796 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.298457    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Nov 02 13:59:40.412698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.407852    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 13:59:40.412698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.407893    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"]
Nov 02 13:59:40.412698 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.408003    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.468589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.457087    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7tv7\" (UniqueName: \"kubernetes.io/projected/d9ff0c33-61e8-4dee-8014-411f7b19389e-kube-api-access-n7tv7\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.468589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.457335    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.468589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.457517    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.468589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.462104    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:59:40.521064 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.513086    5572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:40.570486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.570457    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:59:40.584504 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.584476    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2"
Nov 02 13:59:40.655001 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.654955    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 02 13:59:40.699438 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.698990    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:59:40.727667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.701880    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:40.727667 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.703470    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 13:59:40.728839 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.728734    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:59:40.766067 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.766035    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:59:40.817597 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:40.817436    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4a77d6_bd5b_435d_b57c_fe6cea575bd5.slice/crio-15c17c3f9dca60c5a5cfa2fe6d7b924555338fa2ade45afc849f477ea8fddf02 WatchSource:0}: Error finding container 15c17c3f9dca60c5a5cfa2fe6d7b924555338fa2ade45afc849f477ea8fddf02: Status 404 returned error can't find the container with id 15c17c3f9dca60c5a5cfa2fe6d7b924555338fa2ade45afc849f477ea8fddf02
Nov 02 13:59:40.829031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.827877    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:59:40.829031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.828072    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:59:40.829031 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.828163    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 13:59:40.831714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.831680    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.831918 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.831897    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7tv7\" (UniqueName: \"kubernetes.io/projected/d9ff0c33-61e8-4dee-8014-411f7b19389e-kube-api-access-n7tv7\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.833461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.832087    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.834306 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.834283    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.842639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.839447    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.848725 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.848544    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:59:40.867002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.866959    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7tv7\" (UniqueName: \"kubernetes.io/projected/d9ff0c33-61e8-4dee-8014-411f7b19389e-kube-api-access-n7tv7\") pod \"maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:40.880071 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.878132    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 13:59:40.895467 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.882356    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 13:59:40.895467 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.882408    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:59:40.895467 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.882475    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 13:59:40.972953 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.972904    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"]
Nov 02 13:59:41.001499 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:40.999088    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"]
Nov 02 13:59:41.039706 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.039302    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"]
Nov 02 13:59:41.094868 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.094786    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:41.130368 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.130340    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:41.138010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.136327    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-xks-ca\") pod \"996429a6-9cfc-4a67-9905-02c678287bb5\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") "
Nov 02 13:59:41.138010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.136476    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996429a6-9cfc-4a67-9905-02c678287bb5-kube-api-access-zl5m7\" (UniqueName: \"kubernetes.io/projected/996429a6-9cfc-4a67-9905-02c678287bb5-kube-api-access-zl5m7\") pod \"996429a6-9cfc-4a67-9905-02c678287bb5\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") "
Nov 02 13:59:41.138010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.136775    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-openshift-ca\") pod \"996429a6-9cfc-4a67-9905-02c678287bb5\" (UID: \"996429a6-9cfc-4a67-9905-02c678287bb5\") "
Nov 02 13:59:41.138367 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.138062    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-openshift-ca" pod "996429a6-9cfc-4a67-9905-02c678287bb5" (UID: "996429a6-9cfc-4a67-9905-02c678287bb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:41.140680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.140638    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996429a6-9cfc-4a67-9905-02c678287bb5-kube-api-access-zl5m7" pod "996429a6-9cfc-4a67-9905-02c678287bb5" (UID: "996429a6-9cfc-4a67-9905-02c678287bb5"). InnerVolumeSpecName "kube-api-access-zl5m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:41.145982 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.145959    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-xks-ca" pod "996429a6-9cfc-4a67-9905-02c678287bb5" (UID: "996429a6-9cfc-4a67-9905-02c678287bb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:41.249613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.249548    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl5m7\" (UniqueName: \"kubernetes.io/projected/996429a6-9cfc-4a67-9905-02c678287bb5-kube-api-access-zl5m7\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:41.249825 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.249810    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:41.249937 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.249915    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996429a6-9cfc-4a67-9905-02c678287bb5-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:41.372600 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.368523    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" containerID="cri-o://5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec" gracePeriod=45
Nov 02 13:59:41.383485 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.374157    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" containerID="cri-o://2f40fdf3ab5d615dafbfc3e753fb3a1be6c1655c59a777f09c02468e5c153e45" gracePeriod=45
Nov 02 13:59:41.385896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.385821    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" containerID="cri-o://cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c" gracePeriod=45
Nov 02 13:59:41.426985 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.426934    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": read tcp 10.134.0.2:38468->10.134.0.50:8000: read: connection reset by peer"
Nov 02 13:59:41.448340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.445325    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"
Nov 02 13:59:41.448340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.445351    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw" event={"ID":"996429a6-9cfc-4a67-9905-02c678287bb5","Type":"ContainerDied","Data":"67e9a2ddc4006dbd3b00233c02ada7c1f52e2eddc60331163fde8764a08e86af"}
Nov 02 13:59:41.448340 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.445410    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67e9a2ddc4006dbd3b00233c02ada7c1f52e2eddc60331163fde8764a08e86af"
Nov 02 13:59:41.449867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.449612    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" event={"ID":"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5","Type":"ContainerStarted","Data":"fb0350525175bab04b17f91ce61bd1c6b008d75b38305627db50ab2c1c4bcbb7"}
Nov 02 13:59:41.449867 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.449667    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" event={"ID":"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5","Type":"ContainerStarted","Data":"15c17c3f9dca60c5a5cfa2fe6d7b924555338fa2ade45afc849f477ea8fddf02"}
Nov 02 13:59:41.474301 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.474252    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podStartSLOduration=5.474237062 podStartE2EDuration="5.474237062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:41.472941193 +0000 UTC m=+958.051030073" watchObservedRunningTime="2026-11-02 13:59:41.474237062 +0000 UTC m=+958.052325930"
Nov 02 13:59:41.515026 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.514983    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"]
Nov 02 13:59:42.055584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.980837    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:42.055584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.981715    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:42.055584 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:41.981762    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:42.566668 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.566358    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"]
Nov 02 13:59:42.568621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.568598    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:59:42.571245 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.571220    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:59:42.573825 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.573732    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:59:42.600631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.580166    5572 generic.go:356] "Generic (PLEG): container finished" podID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerID="cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c" exitCode=0
Nov 02 13:59:42.600631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.580319    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" event={"ID":"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e","Type":"ContainerDied","Data":"cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c"}
Nov 02 13:59:42.600631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.580353    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" event={"ID":"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e","Type":"ContainerDied","Data":"fbf49cf0e01dd77573eb63bab30cfd08fe2a766459274449a1ee9a9907e514b5"}
Nov 02 13:59:42.600631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.580375    5572 scope.go:122] "RemoveContainer" containerID="cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c"
Nov 02 13:59:42.600631 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.580512    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"
Nov 02 13:59:42.615948 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.615915    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"]
Nov 02 13:59:42.626966 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.625817    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x" event={"ID":"d9ff0c33-61e8-4dee-8014-411f7b19389e","Type":"ContainerStarted","Data":"bb7a71f9bf732397aeba0746811dc69d691b07ae110a43499e7cb16283653f94"}
Nov 02 13:59:42.630818 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.630731    5572 generic.go:356] "Generic (PLEG): container finished" podID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerID="2f40fdf3ab5d615dafbfc3e753fb3a1be6c1655c59a777f09c02468e5c153e45" exitCode=0
Nov 02 13:59:42.631073 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.630971    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"
Nov 02 13:59:42.631237 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.631087    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" event={"ID":"6c9c6a65-8430-434d-bc56-61b9afec9731","Type":"ContainerDied","Data":"2f40fdf3ab5d615dafbfc3e753fb3a1be6c1655c59a777f09c02468e5c153e45"}
Nov 02 13:59:42.636480 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.636452    5572 generic.go:356] "Generic (PLEG): container finished" podID="d6305498-2daa-47b9-b2a4-e6febb655020" containerID="5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec" exitCode=0
Nov 02 13:59:42.636950 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.636932    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"
Nov 02 13:59:42.646402 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.643251    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" event={"ID":"d6305498-2daa-47b9-b2a4-e6febb655020","Type":"ContainerDied","Data":"5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec"}
Nov 02 13:59:42.646626 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.646601    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" event={"ID":"d6305498-2daa-47b9-b2a4-e6febb655020","Type":"ContainerDied","Data":"749b82b2548378fdf2167f6fff47abbf4469537248b97a506790aa7b4b7ea35a"}
Nov 02 13:59:42.649999 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.649982    5572 scope.go:122] "RemoveContainer" containerID="cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c"
Nov 02 13:59:42.650506 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:42.650483    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c\": container with ID starting with cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c not found: ID does not exist" containerID="cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c"
Nov 02 13:59:42.652594 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.652548    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c"} err="failed to get container status \"cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c\": rpc error: code = NotFound desc = could not find container \"cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c\": container with ID starting with cc33e7f008e11b5c10bd32a086defb6698085e78e195da96d10ebb16fffe637c not found: ID does not exist"
Nov 02 13:59:42.652731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.652718    5572 scope.go:122] "RemoveContainer" containerID="2f40fdf3ab5d615dafbfc3e753fb3a1be6c1655c59a777f09c02468e5c153e45"
Nov 02 13:59:42.721383 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.721354    5572 scope.go:122] "RemoveContainer" containerID="5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec"
Nov 02 13:59:42.731076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731045    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tls-certs\") pod \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") "
Nov 02 13:59:42.731488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731144    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-model-cache\") pod \"6c9c6a65-8430-434d-bc56-61b9afec9731\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") "
Nov 02 13:59:42.731488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731211    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6305498-2daa-47b9-b2a4-e6febb655020-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6305498-2daa-47b9-b2a4-e6febb655020-tls-certs\") pod \"d6305498-2daa-47b9-b2a4-e6febb655020\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") "
Nov 02 13:59:42.731488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731251    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-tmp-dir\") pod \"d6305498-2daa-47b9-b2a4-e6febb655020\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") "
Nov 02 13:59:42.731488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731293    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9c6a65-8430-434d-bc56-61b9afec9731-kube-api-access-bdswp\" (UniqueName: \"kubernetes.io/projected/6c9c6a65-8430-434d-bc56-61b9afec9731-kube-api-access-bdswp\") pod \"6c9c6a65-8430-434d-bc56-61b9afec9731\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") "
Nov 02 13:59:42.731488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731337    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-model-cache\") pod \"d6305498-2daa-47b9-b2a4-e6febb655020\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") "
Nov 02 13:59:42.731488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731382    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9c6a65-8430-434d-bc56-61b9afec9731-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9c6a65-8430-434d-bc56-61b9afec9731-tls-certs\") pod \"6c9c6a65-8430-434d-bc56-61b9afec9731\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") "
Nov 02 13:59:42.731488 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731415    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6305498-2daa-47b9-b2a4-e6febb655020-kube-api-access-97kdt\" (UniqueName: \"kubernetes.io/projected/d6305498-2daa-47b9-b2a4-e6febb655020-kube-api-access-97kdt\") pod \"d6305498-2daa-47b9-b2a4-e6febb655020\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") "
Nov 02 13:59:42.731946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731495    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-model-cache\") pod \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") "
Nov 02 13:59:42.731946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731576    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-kube-api-access-4ll4n\" (UniqueName: \"kubernetes.io/projected/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-kube-api-access-4ll4n\") pod \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") "
Nov 02 13:59:42.731946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731606    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-home\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-home\") pod \"6c9c6a65-8430-434d-bc56-61b9afec9731\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") "
Nov 02 13:59:42.731946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731632    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-dshm\") pod \"d6305498-2daa-47b9-b2a4-e6febb655020\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") "
Nov 02 13:59:42.731946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731665    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tmp-dir\") pod \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") "
Nov 02 13:59:42.731946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731710    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-dshm\") pod \"6c9c6a65-8430-434d-bc56-61b9afec9731\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") "
Nov 02 13:59:42.731946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.731763    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-tmp-dir" pod "d6305498-2daa-47b9-b2a4-e6febb655020" (UID: "d6305498-2daa-47b9-b2a4-e6febb655020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.732673 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.732651    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-model-cache" pod "6c9c6a65-8430-434d-bc56-61b9afec9731" (UID: "6c9c6a65-8430-434d-bc56-61b9afec9731"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.733214 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.733190    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-tmp-dir\") pod \"6c9c6a65-8430-434d-bc56-61b9afec9731\" (UID: \"6c9c6a65-8430-434d-bc56-61b9afec9731\") "
Nov 02 13:59:42.733419 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.733351    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-home\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-home\") pod \"d6305498-2daa-47b9-b2a4-e6febb655020\" (UID: \"d6305498-2daa-47b9-b2a4-e6febb655020\") "
Nov 02 13:59:42.733607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.733539    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-home\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-home\") pod \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") "
Nov 02 13:59:42.733607 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.733579    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-model-cache" pod "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" (UID: "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.733817 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.733795    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-home" pod "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" (UID: "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.733988 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.733969    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-tmp-dir" pod "6c9c6a65-8430-434d-bc56-61b9afec9731" (UID: "6c9c6a65-8430-434d-bc56-61b9afec9731"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.734306 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.734288    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-home" pod "d6305498-2daa-47b9-b2a4-e6febb655020" (UID: "d6305498-2daa-47b9-b2a4-e6febb655020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.734620 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.734601    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-home" pod "6c9c6a65-8430-434d-bc56-61b9afec9731" (UID: "6c9c6a65-8430-434d-bc56-61b9afec9731"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.735820 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.735791    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tls-certs" pod "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" (UID: "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:42.735917 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.735861    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-dshm" pod "6c9c6a65-8430-434d-bc56-61b9afec9731" (UID: "6c9c6a65-8430-434d-bc56-61b9afec9731"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.736109 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.736089    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tmp-dir" pod "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" (UID: "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.737997 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.737696    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9c6a65-8430-434d-bc56-61b9afec9731-kube-api-access-bdswp" pod "6c9c6a65-8430-434d-bc56-61b9afec9731" (UID: "6c9c6a65-8430-434d-bc56-61b9afec9731"). InnerVolumeSpecName "kube-api-access-bdswp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:42.738300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.738276    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-kube-api-access-4ll4n" pod "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" (UID: "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e"). InnerVolumeSpecName "kube-api-access-4ll4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:42.738765 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.738745    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-dshm\") pod \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\" (UID: \"3ef2134c-0ee4-4cb3-9f34-8d42f87c154e\") "
Nov 02 13:59:42.739344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.739296    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9c6a65-8430-434d-bc56-61b9afec9731-tls-certs" pod "6c9c6a65-8430-434d-bc56-61b9afec9731" (UID: "6c9c6a65-8430-434d-bc56-61b9afec9731"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:42.739968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.739531    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6305498-2daa-47b9-b2a4-e6febb655020-tls-certs" pod "d6305498-2daa-47b9-b2a4-e6febb655020" (UID: "d6305498-2daa-47b9-b2a4-e6febb655020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:42.739968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.739611    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-model-cache" pod "d6305498-2daa-47b9-b2a4-e6febb655020" (UID: "d6305498-2daa-47b9-b2a4-e6febb655020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.739968 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.739809    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-dshm" pod "d6305498-2daa-47b9-b2a4-e6febb655020" (UID: "d6305498-2daa-47b9-b2a4-e6febb655020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.740304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.740207    5572 scope.go:122] "RemoveContainer" containerID="5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec"
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:42.740758    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec\": container with ID starting with 5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec not found: ID does not exist" containerID="5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec"
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.740797    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec"} err="failed to get container status \"5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec\": rpc error: code = NotFound desc = could not find container \"5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec\": container with ID starting with 5de49dd92f083a54a2addc76679bcbb0c62716644fc55f9828ee3fd011ac98ec not found: ID does not exist"
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741236    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741262    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741277    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741292    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741306    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741320    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6305498-2daa-47b9-b2a4-e6febb655020-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741334    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741350    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdswp\" (UniqueName: \"kubernetes.io/projected/6c9c6a65-8430-434d-bc56-61b9afec9731-kube-api-access-bdswp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741365    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741379    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c9c6a65-8430-434d-bc56-61b9afec9731-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741394    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741410    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ll4n\" (UniqueName: \"kubernetes.io/projected/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-kube-api-access-4ll4n\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741423    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741435    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6305498-2daa-47b9-b2a4-e6febb655020-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741448    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741459    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c9c6a65-8430-434d-bc56-61b9afec9731-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.741944 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.741776    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-dshm" pod "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" (UID: "3ef2134c-0ee4-4cb3-9f34-8d42f87c154e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:42.743729 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.743709    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6305498-2daa-47b9-b2a4-e6febb655020-kube-api-access-97kdt" pod "d6305498-2daa-47b9-b2a4-e6febb655020" (UID: "d6305498-2daa-47b9-b2a4-e6febb655020"). InnerVolumeSpecName "kube-api-access-97kdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:42.772431 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.772367    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x" podStartSLOduration=3.772346147 podStartE2EDuration="3.772346147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:42.761883112 +0000 UTC m=+959.339971981" watchObservedRunningTime="2026-11-02 13:59:42.772346147 +0000 UTC m=+959.350435015"
Nov 02 13:59:42.842513 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.842483    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.842683 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.842519    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kdt\" (UniqueName: \"kubernetes.io/projected/d6305498-2daa-47b9-b2a4-e6febb655020-kube-api-access-97kdt\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:42.961413 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.961358    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"]
Nov 02 13:59:42.965802 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.965776    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx"]
Nov 02 13:59:42.975863 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.975797    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"]
Nov 02 13:59:42.978131 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.978104    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:42.978251 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:42.978158    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:43.010649 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.010619    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx"]
Nov 02 13:59:43.124654 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.124618    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"]
Nov 02 13:59:43.154825 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.154779    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-2-w6-1eb567-kserve-75f6f46bd4-7lnbx" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:43.187285 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.187247    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455"]
Nov 02 13:59:43.263213 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.263106    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/distinct-w6-1eb567-kserve-575547f587-jlnsx" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:43.470603 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.470350    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1eb567/premium-w6-1eb567-kserve-54859757dd-29455" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:43.626377 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.626345    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" path="/var/lib/kubelet/pods/3ef2134c-0ee4-4cb3-9f34-8d42f87c154e/volumes"
Nov 02 13:59:43.627341 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.627298    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" path="/var/lib/kubelet/pods/6c9c6a65-8430-434d-bc56-61b9afec9731/volumes"
Nov 02 13:59:43.628074 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.628055    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" path="/var/lib/kubelet/pods/d6305498-2daa-47b9-b2a4-e6febb655020/volumes"
Nov 02 13:59:43.658161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.657512    5572 generic.go:356] "Generic (PLEG): container finished" podID="d9ff0c33-61e8-4dee-8014-411f7b19389e" containerID="4399a989ed4c9854d4c379bf4d87768c78278d754a8bb908d3b816b204c93c9d" exitCode=0
Nov 02 13:59:43.658161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.657724    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x" event={"ID":"d9ff0c33-61e8-4dee-8014-411f7b19389e","Type":"ContainerDied","Data":"4399a989ed4c9854d4c379bf4d87768c78278d754a8bb908d3b816b204c93c9d"}
Nov 02 13:59:43.978191 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.978094    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:43.978191 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:43.978165    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:44.147186 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147149    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"]
Nov 02 13:59:44.147908 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147892    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main"
Nov 02 13:59:44.147952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147909    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main"
Nov 02 13:59:44.147952 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147951    5572 cpu_manager.go:414] "containerMap: removing container" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main"
Nov 02 13:59:44.148010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147956    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main"
Nov 02 13:59:44.148010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147965    5572 cpu_manager.go:414] "containerMap: removing container" podUID="996429a6-9cfc-4a67-9905-02c678287bb5" containerName="revoke-keys"
Nov 02 13:59:44.148010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147970    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996429a6-9cfc-4a67-9905-02c678287bb5" containerName="revoke-keys"
Nov 02 13:59:44.148010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147988    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main"
Nov 02 13:59:44.148010 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.147993    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main"
Nov 02 13:59:44.148157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.148080    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9c6a65-8430-434d-bc56-61b9afec9731" containerName="main"
Nov 02 13:59:44.148157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.148088    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="996429a6-9cfc-4a67-9905-02c678287bb5" containerName="revoke-keys"
Nov 02 13:59:44.148157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.148097    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6305498-2daa-47b9-b2a4-e6febb655020" containerName="main"
Nov 02 13:59:44.148157 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.148104    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef2134c-0ee4-4cb3-9f34-8d42f87c154e" containerName="main"
Nov 02 13:59:44.154278 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.154248    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.157939 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.157908    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"]
Nov 02 13:59:44.164689 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.163727    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.166834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.166810    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"]
Nov 02 13:59:44.212974 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.212943    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"]
Nov 02 13:59:44.311809 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.311773    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46gbp\" (UniqueName: \"kubernetes.io/projected/db1d9be0-1e57-478a-89a7-b7a90d53bb39-kube-api-access-46gbp\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.311961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.311824    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52nv\" (UniqueName: \"kubernetes.io/projected/6e1989f9-fef7-4786-987f-1c913e281f90-kube-api-access-l52nv\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.311961 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.311921    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.312136 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.312111    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.312180 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.312160    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.312229 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.312218    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.413401 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.413368    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.413641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.413418    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.413641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.413465    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.413641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.413527    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46gbp\" (UniqueName: \"kubernetes.io/projected/db1d9be0-1e57-478a-89a7-b7a90d53bb39-kube-api-access-46gbp\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.413641 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.413593    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52nv\" (UniqueName: \"kubernetes.io/projected/6e1989f9-fef7-4786-987f-1c913e281f90-kube-api-access-l52nv\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.413876 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.413653    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.414190 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.414151    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.414343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.414325    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.416159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.416130    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.416159 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.416145    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.425962 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.425937    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52nv\" (UniqueName: \"kubernetes.io/projected/6e1989f9-fef7-4786-987f-1c913e281f90-kube-api-access-l52nv\") pod \"maas-api-revoke-sub-2b5a19c8-hhpxd\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.426163 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.426139    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46gbp\" (UniqueName: \"kubernetes.io/projected/db1d9be0-1e57-478a-89a7-b7a90d53bb39-kube-api-access-46gbp\") pod \"maas-api-revoke-sub-53f367ae-sjt2d\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.475151 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.475119    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:44.518098 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.518073    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:44.963596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.963538    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"]
Nov 02 13:59:44.978722 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.978695    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:44.978808 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:44.978752    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:45.032718 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.032685    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:45.047038 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:45.047009    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1d9be0_1e57_478a_89a7_b7a90d53bb39.slice/crio-4cf16b007e09b48a491a0876d82ae6dfe0987f35af23de234515dc35dae0b468 WatchSource:0}: Error finding container 4cf16b007e09b48a491a0876d82ae6dfe0987f35af23de234515dc35dae0b468: Status 404 returned error can't find the container with id 4cf16b007e09b48a491a0876d82ae6dfe0987f35af23de234515dc35dae0b468
Nov 02 13:59:45.049461 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.049438    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"]
Nov 02 13:59:45.075038 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.075005    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"]
Nov 02 13:59:45.076312 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.076093    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d9ff0c33-61e8-4dee-8014-411f7b19389e" containerName="revoke-keys"
Nov 02 13:59:45.076312 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.076118    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ff0c33-61e8-4dee-8014-411f7b19389e" containerName="revoke-keys"
Nov 02 13:59:45.076553 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.076339    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ff0c33-61e8-4dee-8014-411f7b19389e" containerName="revoke-keys"
Nov 02 13:59:45.124859 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.084596    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.128051 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.127984    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"]
Nov 02 13:59:45.140571 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.140519    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"]
Nov 02 13:59:45.144553 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.144521    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ff0c33-61e8-4dee-8014-411f7b19389e-kube-api-access-n7tv7\" (UniqueName: \"kubernetes.io/projected/d9ff0c33-61e8-4dee-8014-411f7b19389e-kube-api-access-n7tv7\") pod \"d9ff0c33-61e8-4dee-8014-411f7b19389e\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") "
Nov 02 13:59:45.145291 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.145272    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-openshift-ca\") pod \"d9ff0c33-61e8-4dee-8014-411f7b19389e\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") "
Nov 02 13:59:45.145646 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.145631    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-xks-ca\") pod \"d9ff0c33-61e8-4dee-8014-411f7b19389e\" (UID: \"d9ff0c33-61e8-4dee-8014-411f7b19389e\") "
Nov 02 13:59:45.145769 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.145740    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-openshift-ca" pod "d9ff0c33-61e8-4dee-8014-411f7b19389e" (UID: "d9ff0c33-61e8-4dee-8014-411f7b19389e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:45.146486 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.146469    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:45.147350 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.147323    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.149790 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.149729    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"]
Nov 02 13:59:45.208852 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.208735    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-xks-ca" pod "d9ff0c33-61e8-4dee-8014-411f7b19389e" (UID: "d9ff0c33-61e8-4dee-8014-411f7b19389e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:45.213202 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.212788    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ff0c33-61e8-4dee-8014-411f7b19389e-kube-api-access-n7tv7" pod "d9ff0c33-61e8-4dee-8014-411f7b19389e" (UID: "d9ff0c33-61e8-4dee-8014-411f7b19389e"). InnerVolumeSpecName "kube-api-access-n7tv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:45.247761 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.247729    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.247926 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.247786    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.247983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.247937    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.247983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.247971    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9gh\" (UniqueName: \"kubernetes.io/projected/20bdaa6b-c6d6-482b-9666-88d140d3057d-kube-api-access-7s9gh\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.248099 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.248007    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.248099 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.248028    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9bj\" (UniqueName: \"kubernetes.io/projected/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-kube-api-access-zs9bj\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.248211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.248173    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ff0c33-61e8-4dee-8014-411f7b19389e-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:45.248211 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.248185    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7tv7\" (UniqueName: \"kubernetes.io/projected/d9ff0c33-61e8-4dee-8014-411f7b19389e-kube-api-access-n7tv7\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:45.349264 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.349218    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.349429 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.349289    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.349429 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.349413    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.349602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.349452    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9gh\" (UniqueName: \"kubernetes.io/projected/20bdaa6b-c6d6-482b-9666-88d140d3057d-kube-api-access-7s9gh\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.349602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.349503    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.349602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.349532    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9bj\" (UniqueName: \"kubernetes.io/projected/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-kube-api-access-zs9bj\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.350078 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.350049    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.350348 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.350204    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.427653 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.427361    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.428058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.428012    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9gh\" (UniqueName: \"kubernetes.io/projected/20bdaa6b-c6d6-482b-9666-88d140d3057d-kube-api-access-7s9gh\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.428660 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.428623    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9bj\" (UniqueName: \"kubernetes.io/projected/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-kube-api-access-zs9bj\") pod \"maas-api-revoke-sub-19350488-nzsqt\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.432806 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.432760    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-718ada8f-799c5\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.436060 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.436034    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:45.472823 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.472790    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:45.833250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.833209    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d" event={"ID":"db1d9be0-1e57-478a-89a7-b7a90d53bb39","Type":"ContainerStarted","Data":"d1f50a9fae125e0e20065bc2633f50ac7173bce847556c15f23038b7c8f8cb6e"}
Nov 02 13:59:45.833538 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.833502    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d" event={"ID":"db1d9be0-1e57-478a-89a7-b7a90d53bb39","Type":"ContainerStarted","Data":"4cf16b007e09b48a491a0876d82ae6dfe0987f35af23de234515dc35dae0b468"}
Nov 02 13:59:45.838210 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.838117    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd" event={"ID":"6e1989f9-fef7-4786-987f-1c913e281f90","Type":"ContainerStarted","Data":"8f48fa744f60785b55424c165d5115554066d1af9ed9b8ff0a8c3b3283ca4c99"}
Nov 02 13:59:45.838403 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.838381    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd" event={"ID":"6e1989f9-fef7-4786-987f-1c913e281f90","Type":"ContainerStarted","Data":"7975d28a6fef30db1b0a97e166c9d5342fd6a3d21521666ace0fc6539974b78e"}
Nov 02 13:59:45.840818 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.840794    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x" event={"ID":"d9ff0c33-61e8-4dee-8014-411f7b19389e","Type":"ContainerDied","Data":"bb7a71f9bf732397aeba0746811dc69d691b07ae110a43499e7cb16283653f94"}
Nov 02 13:59:45.840974 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.840959    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb7a71f9bf732397aeba0746811dc69d691b07ae110a43499e7cb16283653f94"
Nov 02 13:59:45.841141 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.841127    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"
Nov 02 13:59:45.855333 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.855284    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d" podStartSLOduration=1.855268949 podStartE2EDuration="1.855268949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:45.854945194 +0000 UTC m=+962.433034065" watchObservedRunningTime="2026-11-02 13:59:45.855268949 +0000 UTC m=+962.433357819"
Nov 02 13:59:45.914800 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.914737    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd" podStartSLOduration=1.9147164650000001 podStartE2EDuration="1.914716465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:45.880384606 +0000 UTC m=+962.458473475" watchObservedRunningTime="2026-11-02 13:59:45.914716465 +0000 UTC m=+962.492805333"
Nov 02 13:59:45.977666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.977638    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:45.978345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.977825    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:45.979084 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:45.979056    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"]
Nov 02 13:59:46.011728 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.011686    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" podUID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" containerName="payload-processing" containerID="cri-o://526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38" gracePeriod=25
Nov 02 13:59:46.040790 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.040759    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" podUID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" containerName="payload-pre-processing" containerID="cri-o://78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4" gracePeriod=25
Nov 02 13:59:46.157601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.157537    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"]
Nov 02 13:59:46.852169 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.852134    5572 generic.go:356] "Generic (PLEG): container finished" podID="6e1989f9-fef7-4786-987f-1c913e281f90" containerID="8f48fa744f60785b55424c165d5115554066d1af9ed9b8ff0a8c3b3283ca4c99" exitCode=0
Nov 02 13:59:46.852351 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.852335    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd" event={"ID":"6e1989f9-fef7-4786-987f-1c913e281f90","Type":"ContainerDied","Data":"8f48fa744f60785b55424c165d5115554066d1af9ed9b8ff0a8c3b3283ca4c99"}
Nov 02 13:59:46.854199 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.854173    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5" event={"ID":"20bdaa6b-c6d6-482b-9666-88d140d3057d","Type":"ContainerStarted","Data":"3c713db6a82c03d2784fd1efa95a6666ce0e3fcfd6fe2e2be18af59f57e751de"}
Nov 02 13:59:46.854300 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.854217    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5" event={"ID":"20bdaa6b-c6d6-482b-9666-88d140d3057d","Type":"ContainerStarted","Data":"49c43fe3713442a1440d348e765202c22f63e82f534736a938165ea262f0755b"}
Nov 02 13:59:46.856250 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.856227    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt" event={"ID":"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da","Type":"ContainerStarted","Data":"5c09a63028ff1098feb9f5d7319c19445b873df598947603ca2ffba0aebc047d"}
Nov 02 13:59:46.856337 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.856265    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt" event={"ID":"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da","Type":"ContainerStarted","Data":"4e62094bb36b36a8376d877b245e66696adb8f984a93dccbee96da24361c486e"}
Nov 02 13:59:46.858098 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.858078    5572 generic.go:356] "Generic (PLEG): container finished" podID="db1d9be0-1e57-478a-89a7-b7a90d53bb39" containerID="d1f50a9fae125e0e20065bc2633f50ac7173bce847556c15f23038b7c8f8cb6e" exitCode=0
Nov 02 13:59:46.858165 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.858139    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d" event={"ID":"db1d9be0-1e57-478a-89a7-b7a90d53bb39","Type":"ContainerDied","Data":"d1f50a9fae125e0e20065bc2633f50ac7173bce847556c15f23038b7c8f8cb6e"}
Nov 02 13:59:46.934874 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.934809    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt" podStartSLOduration=1.934789031 podStartE2EDuration="1.934789031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:46.930489118 +0000 UTC m=+963.508577987" watchObservedRunningTime="2026-11-02 13:59:46.934789031 +0000 UTC m=+963.512877899"
Nov 02 13:59:46.959637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.959586    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5" podStartSLOduration=1.9595729560000001 podStartE2EDuration="1.959572956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:46.954741709 +0000 UTC m=+963.532830578" watchObservedRunningTime="2026-11-02 13:59:46.959572956 +0000 UTC m=+963.537661821"
Nov 02 13:59:46.977847 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.977813    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:46.978008 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:46.977861    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:47.690589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.687005    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" containerID="cri-o://09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08" gracePeriod=45
Nov 02 13:59:47.736595 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.733741    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" containerID="cri-o://715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d" gracePeriod=45
Nov 02 13:59:47.859343 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.857602    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused"
Nov 02 13:59:47.934588 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.921965    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"]
Nov 02 13:59:47.991205 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.991160    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:47.991443 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.991419    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:47.998170 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.997348    5572 generic.go:356] "Generic (PLEG): container finished" podID="20bdaa6b-c6d6-482b-9666-88d140d3057d" containerID="3c713db6a82c03d2784fd1efa95a6666ce0e3fcfd6fe2e2be18af59f57e751de" exitCode=0
Nov 02 13:59:47.998170 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:47.997620    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5" event={"ID":"20bdaa6b-c6d6-482b-9666-88d140d3057d","Type":"ContainerDied","Data":"3c713db6a82c03d2784fd1efa95a6666ce0e3fcfd6fe2e2be18af59f57e751de"}
Nov 02 13:59:48.016101 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.015194    5572 generic.go:356] "Generic (PLEG): container finished" podID="d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" containerID="5c09a63028ff1098feb9f5d7319c19445b873df598947603ca2ffba0aebc047d" exitCode=0
Nov 02 13:59:48.016101 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.015849    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt" event={"ID":"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da","Type":"ContainerDied","Data":"5c09a63028ff1098feb9f5d7319c19445b873df598947603ca2ffba0aebc047d"}
Nov 02 13:59:48.024246 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.022549    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": EOF"
Nov 02 13:59:48.170941 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.170873    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d18fb-h4k8d"]
Nov 02 13:59:48.511957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.511917    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:48.516233 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.516205    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:48.591770 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.591740    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:59:48.593737 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.593711    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:59:48.601914 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.601880    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-openshift-ca\") pod \"6e1989f9-fef7-4786-987f-1c913e281f90\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") "
Nov 02 13:59:48.602048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.601943    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-xks-ca\") pod \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") "
Nov 02 13:59:48.602048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.601985    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1989f9-fef7-4786-987f-1c913e281f90-kube-api-access-l52nv\" (UniqueName: \"kubernetes.io/projected/6e1989f9-fef7-4786-987f-1c913e281f90-kube-api-access-l52nv\") pod \"6e1989f9-fef7-4786-987f-1c913e281f90\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") "
Nov 02 13:59:48.602048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.602014    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-openshift-ca\") pod \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") "
Nov 02 13:59:48.602280 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.602259    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1d9be0-1e57-478a-89a7-b7a90d53bb39-kube-api-access-46gbp\" (UniqueName: \"kubernetes.io/projected/db1d9be0-1e57-478a-89a7-b7a90d53bb39-kube-api-access-46gbp\") pod \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\" (UID: \"db1d9be0-1e57-478a-89a7-b7a90d53bb39\") "
Nov 02 13:59:48.602442 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.602423    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-xks-ca\") pod \"6e1989f9-fef7-4786-987f-1c913e281f90\" (UID: \"6e1989f9-fef7-4786-987f-1c913e281f90\") "
Nov 02 13:59:48.604175 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.604083    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-openshift-ca" pod "db1d9be0-1e57-478a-89a7-b7a90d53bb39" (UID: "db1d9be0-1e57-478a-89a7-b7a90d53bb39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:48.604609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.604531    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-openshift-ca" pod "6e1989f9-fef7-4786-987f-1c913e281f90" (UID: "6e1989f9-fef7-4786-987f-1c913e281f90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:48.607320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.607239    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1d9be0-1e57-478a-89a7-b7a90d53bb39-kube-api-access-46gbp" pod "db1d9be0-1e57-478a-89a7-b7a90d53bb39" (UID: "db1d9be0-1e57-478a-89a7-b7a90d53bb39"). InnerVolumeSpecName "kube-api-access-46gbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:48.607320 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.607265    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-xks-ca" pod "6e1989f9-fef7-4786-987f-1c913e281f90" (UID: "6e1989f9-fef7-4786-987f-1c913e281f90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:48.612362 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.612321    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-xks-ca" pod "db1d9be0-1e57-478a-89a7-b7a90d53bb39" (UID: "db1d9be0-1e57-478a-89a7-b7a90d53bb39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:48.613217 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.613173    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1989f9-fef7-4786-987f-1c913e281f90-kube-api-access-l52nv" pod "6e1989f9-fef7-4786-987f-1c913e281f90" (UID: "6e1989f9-fef7-4786-987f-1c913e281f90"). InnerVolumeSpecName "kube-api-access-l52nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:48.703753 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.703708    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13cc652-6d83-44e5-bcb0-77481d846a54-tls-certs\" (UniqueName: \"kubernetes.io/secret/e13cc652-6d83-44e5-bcb0-77481d846a54-tls-certs\") pod \"e13cc652-6d83-44e5-bcb0-77481d846a54\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") "
Nov 02 13:59:48.704232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.703784    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34030298-d188-4419-926b-d865e0618a80-kube-api-access-mn5ps\" (UniqueName: \"kubernetes.io/projected/34030298-d188-4419-926b-d865e0618a80-kube-api-access-mn5ps\") pod \"34030298-d188-4419-926b-d865e0618a80\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") "
Nov 02 13:59:48.704232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.703886    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-dshm\") pod \"e13cc652-6d83-44e5-bcb0-77481d846a54\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") "
Nov 02 13:59:48.704232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704020    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13cc652-6d83-44e5-bcb0-77481d846a54-kube-api-access-hbnbg\" (UniqueName: \"kubernetes.io/projected/e13cc652-6d83-44e5-bcb0-77481d846a54-kube-api-access-hbnbg\") pod \"e13cc652-6d83-44e5-bcb0-77481d846a54\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") "
Nov 02 13:59:48.704232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704053    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-tmp-dir\") pod \"e13cc652-6d83-44e5-bcb0-77481d846a54\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") "
Nov 02 13:59:48.704232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704075    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-dshm\") pod \"34030298-d188-4419-926b-d865e0618a80\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") "
Nov 02 13:59:48.704232 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704142    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34030298-d188-4419-926b-d865e0618a80-tls-certs\" (UniqueName: \"kubernetes.io/secret/34030298-d188-4419-926b-d865e0618a80-tls-certs\") pod \"34030298-d188-4419-926b-d865e0618a80\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") "
Nov 02 13:59:48.704643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704286    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-home\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-home\") pod \"34030298-d188-4419-926b-d865e0618a80\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") "
Nov 02 13:59:48.704643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704317    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-home\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-home\") pod \"e13cc652-6d83-44e5-bcb0-77481d846a54\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") "
Nov 02 13:59:48.704643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704350    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-tmp-dir\") pod \"34030298-d188-4419-926b-d865e0618a80\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") "
Nov 02 13:59:48.704643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704378    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-model-cache\") pod \"34030298-d188-4419-926b-d865e0618a80\" (UID: \"34030298-d188-4419-926b-d865e0618a80\") "
Nov 02 13:59:48.704643 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.704454    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-model-cache\") pod \"e13cc652-6d83-44e5-bcb0-77481d846a54\" (UID: \"e13cc652-6d83-44e5-bcb0-77481d846a54\") "
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705456    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705488    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705507    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l52nv\" (UniqueName: \"kubernetes.io/projected/6e1989f9-fef7-4786-987f-1c913e281f90-kube-api-access-l52nv\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705521    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1d9be0-1e57-478a-89a7-b7a90d53bb39-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705536    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46gbp\" (UniqueName: \"kubernetes.io/projected/db1d9be0-1e57-478a-89a7-b7a90d53bb39-kube-api-access-46gbp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705550    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1989f9-fef7-4786-987f-1c913e281f90-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705944    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-tmp-dir" pod "34030298-d188-4419-926b-d865e0618a80" (UID: "34030298-d188-4419-926b-d865e0618a80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.705989    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-tmp-dir" pod "e13cc652-6d83-44e5-bcb0-77481d846a54" (UID: "e13cc652-6d83-44e5-bcb0-77481d846a54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.706159    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-model-cache" pod "34030298-d188-4419-926b-d865e0618a80" (UID: "34030298-d188-4419-926b-d865e0618a80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.706170    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-home" pod "e13cc652-6d83-44e5-bcb0-77481d846a54" (UID: "e13cc652-6d83-44e5-bcb0-77481d846a54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.706508 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.706317    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-model-cache" pod "e13cc652-6d83-44e5-bcb0-77481d846a54" (UID: "e13cc652-6d83-44e5-bcb0-77481d846a54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.707221 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.707196    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-home" pod "34030298-d188-4419-926b-d865e0618a80" (UID: "34030298-d188-4419-926b-d865e0618a80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.707886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.707786    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-dshm" pod "e13cc652-6d83-44e5-bcb0-77481d846a54" (UID: "e13cc652-6d83-44e5-bcb0-77481d846a54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.707886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.707814    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34030298-d188-4419-926b-d865e0618a80-kube-api-access-mn5ps" pod "34030298-d188-4419-926b-d865e0618a80" (UID: "34030298-d188-4419-926b-d865e0618a80"). InnerVolumeSpecName "kube-api-access-mn5ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:48.707886 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.707858    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13cc652-6d83-44e5-bcb0-77481d846a54-tls-certs" pod "e13cc652-6d83-44e5-bcb0-77481d846a54" (UID: "e13cc652-6d83-44e5-bcb0-77481d846a54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:48.709701 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.709677    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-dshm" pod "34030298-d188-4419-926b-d865e0618a80" (UID: "34030298-d188-4419-926b-d865e0618a80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:48.709793 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.709698    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34030298-d188-4419-926b-d865e0618a80-tls-certs" pod "34030298-d188-4419-926b-d865e0618a80" (UID: "34030298-d188-4419-926b-d865e0618a80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:48.709793 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.709716    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13cc652-6d83-44e5-bcb0-77481d846a54-kube-api-access-hbnbg" pod "e13cc652-6d83-44e5-bcb0-77481d846a54" (UID: "e13cc652-6d83-44e5-bcb0-77481d846a54"). InnerVolumeSpecName "kube-api-access-hbnbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:48.807547 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807510    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13cc652-6d83-44e5-bcb0-77481d846a54-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807547 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807544    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn5ps\" (UniqueName: \"kubernetes.io/projected/34030298-d188-4419-926b-d865e0618a80-kube-api-access-mn5ps\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807554    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807604    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbnbg\" (UniqueName: \"kubernetes.io/projected/e13cc652-6d83-44e5-bcb0-77481d846a54-kube-api-access-hbnbg\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807612    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807620    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807630    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34030298-d188-4419-926b-d865e0618a80-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807637    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807645    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807654    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807663    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34030298-d188-4419-926b-d865e0618a80-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.807755 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.807671    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13cc652-6d83-44e5-bcb0-77481d846a54-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:48.978383 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.978347    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 13:59:48.978613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:48.978408    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 13:59:49.026297 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.026260    5572 generic.go:356] "Generic (PLEG): container finished" podID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerID="09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08" exitCode=0
Nov 02 13:59:49.026447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.026304    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"
Nov 02 13:59:49.026447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.026347    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" event={"ID":"e13cc652-6d83-44e5-bcb0-77481d846a54","Type":"ContainerDied","Data":"09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08"}
Nov 02 13:59:49.026447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.026386    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2" event={"ID":"e13cc652-6d83-44e5-bcb0-77481d846a54","Type":"ContainerDied","Data":"1f778eac2ef5c720b975b601127f9a150d1bf72549af4db072b9a61fc9504919"}
Nov 02 13:59:49.026447 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.026408    5572 scope.go:122] "RemoveContainer" containerID="09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08"
Nov 02 13:59:49.028528 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.028493    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d" event={"ID":"db1d9be0-1e57-478a-89a7-b7a90d53bb39","Type":"ContainerDied","Data":"4cf16b007e09b48a491a0876d82ae6dfe0987f35af23de234515dc35dae0b468"}
Nov 02 13:59:49.028692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.028610    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cf16b007e09b48a491a0876d82ae6dfe0987f35af23de234515dc35dae0b468"
Nov 02 13:59:49.028692 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.028680    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"
Nov 02 13:59:49.034289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.033711    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"
Nov 02 13:59:49.034289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.033721    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd" event={"ID":"6e1989f9-fef7-4786-987f-1c913e281f90","Type":"ContainerDied","Data":"7975d28a6fef30db1b0a97e166c9d5342fd6a3d21521666ace0fc6539974b78e"}
Nov 02 13:59:49.034289 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.033850    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7975d28a6fef30db1b0a97e166c9d5342fd6a3d21521666ace0fc6539974b78e"
Nov 02 13:59:49.037304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.036235    5572 generic.go:356] "Generic (PLEG): container finished" podID="34030298-d188-4419-926b-d865e0618a80" containerID="715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d" exitCode=0
Nov 02 13:59:49.037304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.036310    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"
Nov 02 13:59:49.037304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.036341    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" event={"ID":"34030298-d188-4419-926b-d865e0618a80","Type":"ContainerDied","Data":"715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d"}
Nov 02 13:59:49.037304 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.036371    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969" event={"ID":"34030298-d188-4419-926b-d865e0618a80","Type":"ContainerDied","Data":"34fe20b8cdb37a8f9d4388dd7c1ce114ee9b433f38a567a66c8ecc4d5bfb323e"}
Nov 02 13:59:49.055621 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.051118    5572 scope.go:122] "RemoveContainer" containerID="09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08"
Nov 02 13:59:49.056857 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:49.056816    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08\": container with ID starting with 09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08 not found: ID does not exist" containerID="09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08"
Nov 02 13:59:49.056983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.056870    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08"} err="failed to get container status \"09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08\": rpc error: code = NotFound desc = could not find container \"09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08\": container with ID starting with 09adf0acd313c3f3ecbe8d871b34be0bc971df7f4da0976df83f6dd5ff293b08 not found: ID does not exist"
Nov 02 13:59:49.056983 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.056897    5572 scope.go:122] "RemoveContainer" containerID="715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d"
Nov 02 13:59:49.060609 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.058125    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" podUID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:59:49.082030 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.081941    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" podUID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:59:49.088167 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.083251    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"]
Nov 02 13:59:49.090751 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.090715    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/distinct-2-w3-4fc49b-kserve-5b54ddd65f-fwlk2"]
Nov 02 13:59:49.095033 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.095005    5572 scope.go:122] "RemoveContainer" containerID="715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d"
Nov 02 13:59:49.095536 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:49.095510    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d\": container with ID starting with 715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d not found: ID does not exist" containerID="715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d"
Nov 02 13:59:49.095730 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.095701    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d"} err="failed to get container status \"715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d\": rpc error: code = NotFound desc = could not find container \"715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d\": container with ID starting with 715baa325768c359f3033ccd982e2a039aa7406ecc3d4d5e3f8819b487fb8d6d not found: ID does not exist"
Nov 02 13:59:49.110873 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.110774    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"]
Nov 02 13:59:49.120271 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.120245    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4fc49b/simulator-w3-4fc49b-kserve-6944c6d8d8-jj969"]
Nov 02 13:59:49.275269 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.275237    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:49.279612 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.279584    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:49.316102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.316065    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-xks-ca\") pod \"20bdaa6b-c6d6-482b-9666-88d140d3057d\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") "
Nov 02 13:59:49.316302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.316129    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-kube-api-access-zs9bj\" (UniqueName: \"kubernetes.io/projected/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-kube-api-access-zs9bj\") pod \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") "
Nov 02 13:59:49.316302 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.316236    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-openshift-ca\") pod \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") "
Nov 02 13:59:49.316433 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.316394    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bdaa6b-c6d6-482b-9666-88d140d3057d-kube-api-access-7s9gh\" (UniqueName: \"kubernetes.io/projected/20bdaa6b-c6d6-482b-9666-88d140d3057d-kube-api-access-7s9gh\") pod \"20bdaa6b-c6d6-482b-9666-88d140d3057d\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") "
Nov 02 13:59:49.316493 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.316430    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-openshift-ca\") pod \"20bdaa6b-c6d6-482b-9666-88d140d3057d\" (UID: \"20bdaa6b-c6d6-482b-9666-88d140d3057d\") "
Nov 02 13:59:49.316626 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.316603    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-xks-ca\") pod \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\" (UID: \"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da\") "
Nov 02 13:59:49.317185 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.317155    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-openshift-ca" pod "d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" (UID: "d6c3bd16-cf04-462f-bfe7-c5b2a5a318da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:49.317345 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.317222    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-openshift-ca" pod "20bdaa6b-c6d6-482b-9666-88d140d3057d" (UID: "20bdaa6b-c6d6-482b-9666-88d140d3057d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:49.317531 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.317509    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:49.320466 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.320430    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bdaa6b-c6d6-482b-9666-88d140d3057d-kube-api-access-7s9gh" pod "20bdaa6b-c6d6-482b-9666-88d140d3057d" (UID: "20bdaa6b-c6d6-482b-9666-88d140d3057d"). InnerVolumeSpecName "kube-api-access-7s9gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:49.320900 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.320865    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-xks-ca" pod "20bdaa6b-c6d6-482b-9666-88d140d3057d" (UID: "20bdaa6b-c6d6-482b-9666-88d140d3057d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:49.321171 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.321121    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-kube-api-access-zs9bj" pod "d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" (UID: "d6c3bd16-cf04-462f-bfe7-c5b2a5a318da"). InnerVolumeSpecName "kube-api-access-zs9bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:49.322904 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.322840    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-xks-ca" pod "d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" (UID: "d6c3bd16-cf04-462f-bfe7-c5b2a5a318da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:49.419891 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.419852    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s9gh\" (UniqueName: \"kubernetes.io/projected/20bdaa6b-c6d6-482b-9666-88d140d3057d-kube-api-access-7s9gh\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:49.419891 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.419896    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:49.420102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.419914    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:49.420102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.419930    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20bdaa6b-c6d6-482b-9666-88d140d3057d-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:49.420102 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.419945    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs9bj\" (UniqueName: \"kubernetes.io/projected/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da-kube-api-access-zs9bj\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:49.607658 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.607549    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34030298-d188-4419-926b-d865e0618a80" path="/var/lib/kubelet/pods/34030298-d188-4419-926b-d865e0618a80/volumes"
Nov 02 13:59:49.610869 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.610838    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9" path="/var/lib/kubelet/pods/483ae1b4-7f0e-4b7d-8ec0-b55663fc67f9/volumes"
Nov 02 13:59:49.611811 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.611783    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" path="/var/lib/kubelet/pods/e13cc652-6d83-44e5-bcb0-77481d846a54/volumes"
Nov 02 13:59:49.710431 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:49.710381    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"]
Nov 02 13:59:50.113484 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.113445    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt" event={"ID":"d6c3bd16-cf04-462f-bfe7-c5b2a5a318da","Type":"ContainerDied","Data":"4e62094bb36b36a8376d877b245e66696adb8f984a93dccbee96da24361c486e"}
Nov 02 13:59:50.113786 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.113765    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e62094bb36b36a8376d877b245e66696adb8f984a93dccbee96da24361c486e"
Nov 02 13:59:50.113954 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.113942    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"
Nov 02 13:59:50.128915 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.127895    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5" event={"ID":"20bdaa6b-c6d6-482b-9666-88d140d3057d","Type":"ContainerDied","Data":"49c43fe3713442a1440d348e765202c22f63e82f534736a938165ea262f0755b"}
Nov 02 13:59:50.128915 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.127957    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49c43fe3713442a1440d348e765202c22f63e82f534736a938165ea262f0755b"
Nov 02 13:59:50.128915 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.128026    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"
Nov 02 13:59:50.357449 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.357413    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"]
Nov 02 13:59:50.358750 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.358725    5572 cpu_manager.go:414] "containerMap: removing container" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main"
Nov 02 13:59:50.358896 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.358883    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main"
Nov 02 13:59:50.359012 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359002    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" containerName="revoke-keys"
Nov 02 13:59:50.359086 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359076    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" containerName="revoke-keys"
Nov 02 13:59:50.359162 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359154    5572 cpu_manager.go:414] "containerMap: removing container" podUID="20bdaa6b-c6d6-482b-9666-88d140d3057d" containerName="revoke-keys"
Nov 02 13:59:50.359229 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359220    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bdaa6b-c6d6-482b-9666-88d140d3057d" containerName="revoke-keys"
Nov 02 13:59:50.359352 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359342    5572 cpu_manager.go:414] "containerMap: removing container" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main"
Nov 02 13:59:50.359430 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359420    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main"
Nov 02 13:59:50.359498 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359490    5572 cpu_manager.go:414] "containerMap: removing container" podUID="6e1989f9-fef7-4786-987f-1c913e281f90" containerName="revoke-keys"
Nov 02 13:59:50.359583 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359572    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1989f9-fef7-4786-987f-1c913e281f90" containerName="revoke-keys"
Nov 02 13:59:50.359666 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359657    5572 cpu_manager.go:414] "containerMap: removing container" podUID="db1d9be0-1e57-478a-89a7-b7a90d53bb39" containerName="revoke-keys"
Nov 02 13:59:50.359731 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359722    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1d9be0-1e57-478a-89a7-b7a90d53bb39" containerName="revoke-keys"
Nov 02 13:59:50.359997 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.359983    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1d9be0-1e57-478a-89a7-b7a90d53bb39" containerName="revoke-keys"
Nov 02 13:59:50.360095 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.360086    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="34030298-d188-4419-926b-d865e0618a80" containerName="main"
Nov 02 13:59:50.360178 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.360168    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" containerName="revoke-keys"
Nov 02 13:59:50.360258 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.360237    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bdaa6b-c6d6-482b-9666-88d140d3057d" containerName="revoke-keys"
Nov 02 13:59:50.360344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.360335    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13cc652-6d83-44e5-bcb0-77481d846a54" containerName="main"
Nov 02 13:59:50.360412 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.360404    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1989f9-fef7-4786-987f-1c913e281f90" containerName="revoke-keys"
Nov 02 13:59:50.370626 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.369593    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.376329 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.375816    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"]
Nov 02 13:59:50.388227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.388196    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:50.388405 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.388349    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:59:50.400687 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.400658    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.409890 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.400548    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"]
Nov 02 13:59:50.410028 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.409935    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"]
Nov 02 13:59:50.469417 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.469385    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.469624 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.469445    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtk2\" (UniqueName: \"kubernetes.io/projected/197ce710-8c1b-4e8e-9c1a-3e6810517f23-kube-api-access-sxtk2\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.469725 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.469623    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.469785 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.469728    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.469843 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.469825    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjm8\" (UniqueName: \"kubernetes.io/projected/a0a2af9c-a923-48a4-af56-a0b66084ead1-kube-api-access-5zjm8\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.469957 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.469941    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.572268 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.572216    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.572485 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.572323    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtk2\" (UniqueName: \"kubernetes.io/projected/197ce710-8c1b-4e8e-9c1a-3e6810517f23-kube-api-access-sxtk2\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.572485 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.572401    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.572485 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.572481    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.572714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.572594    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjm8\" (UniqueName: \"kubernetes.io/projected/a0a2af9c-a923-48a4-af56-a0b66084ead1-kube-api-access-5zjm8\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.572819 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.572741    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.573307 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.573284    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.573647 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.573625    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.575720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.575691    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.575857 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.575831    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.586434 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.586376    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtk2\" (UniqueName: \"kubernetes.io/projected/197ce710-8c1b-4e8e-9c1a-3e6810517f23-kube-api-access-sxtk2\") pod \"maas-api-revoke-sub-dc0fcb3f-4wd9l\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:50.587740 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.587688    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjm8\" (UniqueName: \"kubernetes.io/projected/a0a2af9c-a923-48a4-af56-a0b66084ead1-kube-api-access-5zjm8\") pod \"maas-api-revoke-sub-e420e1df-xc694\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.700756 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.700661    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:50.727015 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:50.726983    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:51.094855 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.093845    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:59:51.094855 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.093924    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:51.235464 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.235378    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"]
Nov 02 13:59:51.241599 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.241551    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.259818 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.259790    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"]
Nov 02 13:59:51.263928 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:51.263881    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197ce710_8c1b_4e8e_9c1a_3e6810517f23.slice/crio-baa830643cb316d9912b7f75a2e6e3579f06a5a2ea7a2ae09b060d582052b31e WatchSource:0}: Error finding container baa830643cb316d9912b7f75a2e6e3579f06a5a2ea7a2ae09b060d582052b31e: Status 404 returned error can't find the container with id baa830643cb316d9912b7f75a2e6e3579f06a5a2ea7a2ae09b060d582052b31e
Nov 02 13:59:51.268142 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.268122    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"]
Nov 02 13:59:51.356306 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:51.356273    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a2af9c_a923_48a4_af56_a0b66084ead1.slice/crio-61df475539e420f7f5b33a62245a24dbf830c7f71285cecfbf167b0dda5ddfc5 WatchSource:0}: Error finding container 61df475539e420f7f5b33a62245a24dbf830c7f71285cecfbf167b0dda5ddfc5: Status 404 returned error can't find the container with id 61df475539e420f7f5b33a62245a24dbf830c7f71285cecfbf167b0dda5ddfc5
Nov 02 13:59:51.358587 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.358545    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"]
Nov 02 13:59:51.362959 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.362935    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bcz\" (UniqueName: \"kubernetes.io/projected/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-kube-api-access-h9bcz\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.363122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.363103    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.363179 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.363155    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.438433 ip-10-0-128-62 kubenswrapper[5572]: E1102 13:59:51.438396    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 13:59:51.465227 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.465190    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bcz\" (UniqueName: \"kubernetes.io/projected/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-kube-api-access-h9bcz\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.465380 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.465327    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.465380 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.465363    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.466198 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.466172    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.468173 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.468149    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.479023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.479001    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bcz\" (UniqueName: \"kubernetes.io/projected/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-kube-api-access-h9bcz\") pod \"maas-api-revoke-sub-2a9ba7e3-wzxrq\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.497014 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.496935    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:51.498168 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.498145    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:51.499042 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.499015    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 13:59:51.563696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.563660    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:51.916514 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:51.916467    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9972e7d7_30bf_43f5_8b3d_4783d268c9f9.slice/crio-1f1a56881d2072ce5763d01758d2564ab128b2a34dfe27814a7ed7c75275ba24 WatchSource:0}: Error finding container 1f1a56881d2072ce5763d01758d2564ab128b2a34dfe27814a7ed7c75275ba24: Status 404 returned error can't find the container with id 1f1a56881d2072ce5763d01758d2564ab128b2a34dfe27814a7ed7c75275ba24
Nov 02 13:59:51.919601 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:51.919518    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"]
Nov 02 13:59:52.153370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:52.153280    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694" event={"ID":"a0a2af9c-a923-48a4-af56-a0b66084ead1","Type":"ContainerStarted","Data":"dcb5b05d15feae78292dcdb46904cdbe15757ec9464fb03b9df1fbd1b401b28e"}
Nov 02 13:59:52.153370 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:52.153347    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694" event={"ID":"a0a2af9c-a923-48a4-af56-a0b66084ead1","Type":"ContainerStarted","Data":"61df475539e420f7f5b33a62245a24dbf830c7f71285cecfbf167b0dda5ddfc5"}
Nov 02 13:59:52.155122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:52.155096    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l" event={"ID":"197ce710-8c1b-4e8e-9c1a-3e6810517f23","Type":"ContainerStarted","Data":"e636d826a8119f69b78365f6afed8468b1b9e80aab932686b395b05e43c8c831"}
Nov 02 13:59:52.155229 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:52.155138    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l" event={"ID":"197ce710-8c1b-4e8e-9c1a-3e6810517f23","Type":"ContainerStarted","Data":"baa830643cb316d9912b7f75a2e6e3579f06a5a2ea7a2ae09b060d582052b31e"}
Nov 02 13:59:52.156602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:52.156578    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq" event={"ID":"9972e7d7-30bf-43f5-8b3d-4783d268c9f9","Type":"ContainerStarted","Data":"1f1a56881d2072ce5763d01758d2564ab128b2a34dfe27814a7ed7c75275ba24"}
Nov 02 13:59:52.175532 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:52.175486    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694" podStartSLOduration=2.175472649 podStartE2EDuration="2.175472649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:52.170685985 +0000 UTC m=+968.748774873" watchObservedRunningTime="2026-11-02 13:59:52.175472649 +0000 UTC m=+968.753561515"
Nov 02 13:59:52.191037 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:52.190990    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l" podStartSLOduration=2.190976175 podStartE2EDuration="2.190976175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:52.18920466 +0000 UTC m=+968.767293529" watchObservedRunningTime="2026-11-02 13:59:52.190976175 +0000 UTC m=+968.769065068"
Nov 02 13:59:53.167053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:53.166952    5572 generic.go:356] "Generic (PLEG): container finished" podID="a0a2af9c-a923-48a4-af56-a0b66084ead1" containerID="dcb5b05d15feae78292dcdb46904cdbe15757ec9464fb03b9df1fbd1b401b28e" exitCode=0
Nov 02 13:59:53.167440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:53.167046    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694" event={"ID":"a0a2af9c-a923-48a4-af56-a0b66084ead1","Type":"ContainerDied","Data":"dcb5b05d15feae78292dcdb46904cdbe15757ec9464fb03b9df1fbd1b401b28e"}
Nov 02 13:59:53.168828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:53.168800    5572 generic.go:356] "Generic (PLEG): container finished" podID="197ce710-8c1b-4e8e-9c1a-3e6810517f23" containerID="e636d826a8119f69b78365f6afed8468b1b9e80aab932686b395b05e43c8c831" exitCode=0
Nov 02 13:59:53.168946 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:53.168882    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l" event={"ID":"197ce710-8c1b-4e8e-9c1a-3e6810517f23","Type":"ContainerDied","Data":"e636d826a8119f69b78365f6afed8468b1b9e80aab932686b395b05e43c8c831"}
Nov 02 13:59:53.170624 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:53.170592    5572 generic.go:356] "Generic (PLEG): container finished" podID="9972e7d7-30bf-43f5-8b3d-4783d268c9f9" containerID="f0cd4bcca44e197609ae37b5b5c7f62cd245c5116a7d45e184202e2d1d435715" exitCode=0
Nov 02 13:59:53.170720 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:53.170657    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq" event={"ID":"9972e7d7-30bf-43f5-8b3d-4783d268c9f9","Type":"ContainerDied","Data":"f0cd4bcca44e197609ae37b5b5c7f62cd245c5116a7d45e184202e2d1d435715"}
Nov 02 13:59:54.384192 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.384161    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:54.507030 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.489749    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:54.553596 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.553380    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:54.566396 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.566362    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-xks-ca\") pod \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") "
Nov 02 13:59:54.566662 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.566638    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-openshift-ca\") pod \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") "
Nov 02 13:59:54.566813 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.566689    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197ce710-8c1b-4e8e-9c1a-3e6810517f23-kube-api-access-sxtk2\" (UniqueName: \"kubernetes.io/projected/197ce710-8c1b-4e8e-9c1a-3e6810517f23-kube-api-access-sxtk2\") pod \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\" (UID: \"197ce710-8c1b-4e8e-9c1a-3e6810517f23\") "
Nov 02 13:59:54.567115 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.567085    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-openshift-ca" pod "197ce710-8c1b-4e8e-9c1a-3e6810517f23" (UID: "197ce710-8c1b-4e8e-9c1a-3e6810517f23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:54.567406 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.567385    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.569552 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.569529    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197ce710-8c1b-4e8e-9c1a-3e6810517f23-kube-api-access-sxtk2" pod "197ce710-8c1b-4e8e-9c1a-3e6810517f23" (UID: "197ce710-8c1b-4e8e-9c1a-3e6810517f23"). InnerVolumeSpecName "kube-api-access-sxtk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:54.569790 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.569774    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-xks-ca" pod "197ce710-8c1b-4e8e-9c1a-3e6810517f23" (UID: "197ce710-8c1b-4e8e-9c1a-3e6810517f23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:54.667977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.667930    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-xks-ca\") pod \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") "
Nov 02 13:59:54.668161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.668014    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-openshift-ca\") pod \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") "
Nov 02 13:59:54.668161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.668076    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-kube-api-access-h9bcz\" (UniqueName: \"kubernetes.io/projected/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-kube-api-access-h9bcz\") pod \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\" (UID: \"9972e7d7-30bf-43f5-8b3d-4783d268c9f9\") "
Nov 02 13:59:54.668161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.668107    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a2af9c-a923-48a4-af56-a0b66084ead1-kube-api-access-5zjm8\" (UniqueName: \"kubernetes.io/projected/a0a2af9c-a923-48a4-af56-a0b66084ead1-kube-api-access-5zjm8\") pod \"a0a2af9c-a923-48a4-af56-a0b66084ead1\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") "
Nov 02 13:59:54.668161 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.668146    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-xks-ca\") pod \"a0a2af9c-a923-48a4-af56-a0b66084ead1\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") "
Nov 02 13:59:54.668375 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.668264    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-openshift-ca\") pod \"a0a2af9c-a923-48a4-af56-a0b66084ead1\" (UID: \"a0a2af9c-a923-48a4-af56-a0b66084ead1\") "
Nov 02 13:59:54.668439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.668408    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-openshift-ca" pod "9972e7d7-30bf-43f5-8b3d-4783d268c9f9" (UID: "9972e7d7-30bf-43f5-8b3d-4783d268c9f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:54.669002 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.668905    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-openshift-ca" pod "a0a2af9c-a923-48a4-af56-a0b66084ead1" (UID: "a0a2af9c-a923-48a4-af56-a0b66084ead1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:54.669727 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.669701    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.669860 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.669733    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197ce710-8c1b-4e8e-9c1a-3e6810517f23-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.669860 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.669749    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.669860 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.669763    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtk2\" (UniqueName: \"kubernetes.io/projected/197ce710-8c1b-4e8e-9c1a-3e6810517f23-kube-api-access-sxtk2\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.670410 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.670385    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-xks-ca" pod "9972e7d7-30bf-43f5-8b3d-4783d268c9f9" (UID: "9972e7d7-30bf-43f5-8b3d-4783d268c9f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:54.670538 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.670523    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-xks-ca" pod "a0a2af9c-a923-48a4-af56-a0b66084ead1" (UID: "a0a2af9c-a923-48a4-af56-a0b66084ead1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:54.670613 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.670593    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a2af9c-a923-48a4-af56-a0b66084ead1-kube-api-access-5zjm8" pod "a0a2af9c-a923-48a4-af56-a0b66084ead1" (UID: "a0a2af9c-a923-48a4-af56-a0b66084ead1"). InnerVolumeSpecName "kube-api-access-5zjm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:54.670714 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.670698    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-kube-api-access-h9bcz" pod "9972e7d7-30bf-43f5-8b3d-4783d268c9f9" (UID: "9972e7d7-30bf-43f5-8b3d-4783d268c9f9"). InnerVolumeSpecName "kube-api-access-h9bcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:54.711690 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.711605    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" podUID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" containerName="payload-pre-processing" containerID="cri-o://e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8" gracePeriod=25
Nov 02 13:59:54.770828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.770781    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.770828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.770812    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9bcz\" (UniqueName: \"kubernetes.io/projected/9972e7d7-30bf-43f5-8b3d-4783d268c9f9-kube-api-access-h9bcz\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.770828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.770822    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjm8\" (UniqueName: \"kubernetes.io/projected/a0a2af9c-a923-48a4-af56-a0b66084ead1-kube-api-access-5zjm8\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:54.770828 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:54.770831    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2af9c-a923-48a4-af56-a0b66084ead1-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:55.105189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.100511    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"]
Nov 02 13:59:55.105189 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.100797    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" containerID="cri-o://6b829521a0fa31b14a586d84abe04daabcef0ad5564b38d3fd9f8994f7dde93c" gracePeriod=30
Nov 02 13:59:55.120637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.118292    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"]
Nov 02 13:59:55.120637 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.118528    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" containerID="cri-o://cb594dfef9f4b3f42467f53a68b97dea043d455554d854a72a83ce805c05801d" gracePeriod=30
Nov 02 13:59:55.126006 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.125979    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"]
Nov 02 13:59:55.126190 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.126169    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" containerID="cri-o://af1904c4b7ce1bf0ea416493235f825ce3a7c3470720ae0f0a2ebe26a350192d" gracePeriod=30
Nov 02 13:59:55.189201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.189168    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"
Nov 02 13:59:55.189201 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.189178    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq" event={"ID":"9972e7d7-30bf-43f5-8b3d-4783d268c9f9","Type":"ContainerDied","Data":"1f1a56881d2072ce5763d01758d2564ab128b2a34dfe27814a7ed7c75275ba24"}
Nov 02 13:59:55.189435 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.189258    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f1a56881d2072ce5763d01758d2564ab128b2a34dfe27814a7ed7c75275ba24"
Nov 02 13:59:55.191194 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.191165    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694" event={"ID":"a0a2af9c-a923-48a4-af56-a0b66084ead1","Type":"ContainerDied","Data":"61df475539e420f7f5b33a62245a24dbf830c7f71285cecfbf167b0dda5ddfc5"}
Nov 02 13:59:55.191344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.191217    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61df475539e420f7f5b33a62245a24dbf830c7f71285cecfbf167b0dda5ddfc5"
Nov 02 13:59:55.191344 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.191180    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"
Nov 02 13:59:55.192745 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.192722    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l" event={"ID":"197ce710-8c1b-4e8e-9c1a-3e6810517f23","Type":"ContainerDied","Data":"baa830643cb316d9912b7f75a2e6e3579f06a5a2ea7a2ae09b060d582052b31e"}
Nov 02 13:59:55.192745 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.192742    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"
Nov 02 13:59:55.192931 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:55.192759    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa830643cb316d9912b7f75a2e6e3579f06a5a2ea7a2ae09b060d582052b31e"
Nov 02 13:59:57.102092 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102057    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"]
Nov 02 13:59:57.102779 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102763    5572 cpu_manager.go:414] "containerMap: removing container" podUID="a0a2af9c-a923-48a4-af56-a0b66084ead1" containerName="revoke-keys"
Nov 02 13:59:57.102834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102780    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a2af9c-a923-48a4-af56-a0b66084ead1" containerName="revoke-keys"
Nov 02 13:59:57.102834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102802    5572 cpu_manager.go:414] "containerMap: removing container" podUID="9972e7d7-30bf-43f5-8b3d-4783d268c9f9" containerName="revoke-keys"
Nov 02 13:59:57.102834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102807    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9972e7d7-30bf-43f5-8b3d-4783d268c9f9" containerName="revoke-keys"
Nov 02 13:59:57.102834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102822    5572 cpu_manager.go:414] "containerMap: removing container" podUID="197ce710-8c1b-4e8e-9c1a-3e6810517f23" containerName="revoke-keys"
Nov 02 13:59:57.102965 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102837    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197ce710-8c1b-4e8e-9c1a-3e6810517f23" containerName="revoke-keys"
Nov 02 13:59:57.102965 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102945    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9972e7d7-30bf-43f5-8b3d-4783d268c9f9" containerName="revoke-keys"
Nov 02 13:59:57.102965 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102955    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="197ce710-8c1b-4e8e-9c1a-3e6810517f23" containerName="revoke-keys"
Nov 02 13:59:57.103058 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.102967    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a2af9c-a923-48a4-af56-a0b66084ead1" containerName="revoke-keys"
Nov 02 13:59:57.106977 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.106961    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.111768 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.111701    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:57.112602 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.112521    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 13:59:57.118737 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.118582    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"]
Nov 02 13:59:57.175798 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.175752    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9fj\" (UniqueName: \"kubernetes.io/projected/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-kube-api-access-xk9fj\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.176000 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.175930    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.176147 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.176123    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.277924 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.277882    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9fj\" (UniqueName: \"kubernetes.io/projected/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-kube-api-access-xk9fj\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.278122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.277965    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.278122 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.278057    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.278913 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.278866    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.280616 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.280589    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.295108 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.295082    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9fj\" (UniqueName: \"kubernetes.io/projected/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-kube-api-access-xk9fj\") pod \"maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.434741 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.434660    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 13:59:57.635476 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.635431    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" containerID="cri-o://0ed31b4b7e74cf4e696d942d7b15088cfbe4516deac37b907e84b07accb435b8" gracePeriod=45
Nov 02 13:59:57.677837 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.677750    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" containerID="cri-o://15399b7e8076dfdc465b07db73f51e324bc6d32e5be6478ab7443ca1b681a249" gracePeriod=45
Nov 02 13:59:57.704453 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.704176    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": dial tcp 10.134.0.51:8000: connect: connection refused"
Nov 02 13:59:57.738248 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.738211    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"]
Nov 02 13:59:57.753113 ip-10-0-128-62 kubenswrapper[5572]: W1102 13:59:57.753045    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2659dec_bd6d_4c65_af4c_dbd36bb46cd6.slice/crio-9edb62821d28b08fdeba2066b98a6820cc26e655bc4f8de07ba5554cb0d6df75 WatchSource:0}: Error finding container 9edb62821d28b08fdeba2066b98a6820cc26e655bc4f8de07ba5554cb0d6df75: Status 404 returned error can't find the container with id 9edb62821d28b08fdeba2066b98a6820cc26e655bc4f8de07ba5554cb0d6df75
Nov 02 13:59:57.945186 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:57.940359    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused"
Nov 02 13:59:58.261680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.261298    5572 generic.go:356] "Generic (PLEG): container finished" podID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerID="0ed31b4b7e74cf4e696d942d7b15088cfbe4516deac37b907e84b07accb435b8" exitCode=0
Nov 02 13:59:58.261680 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.261595    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" event={"ID":"d3ff03bf-bb7f-4611-8a37-b9533e028dd8","Type":"ContainerDied","Data":"0ed31b4b7e74cf4e696d942d7b15088cfbe4516deac37b907e84b07accb435b8"}
Nov 02 13:59:58.265053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.264693    5572 generic.go:356] "Generic (PLEG): container finished" podID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerID="15399b7e8076dfdc465b07db73f51e324bc6d32e5be6478ab7443ca1b681a249" exitCode=0
Nov 02 13:59:58.265053 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.264978    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" event={"ID":"dc5fe786-8a48-4a93-9be5-e2092ee937a5","Type":"ContainerDied","Data":"15399b7e8076dfdc465b07db73f51e324bc6d32e5be6478ab7443ca1b681a249"}
Nov 02 13:59:58.269048 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.268964    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf" event={"ID":"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6","Type":"ContainerStarted","Data":"9edb62821d28b08fdeba2066b98a6820cc26e655bc4f8de07ba5554cb0d6df75"}
Nov 02 13:59:58.758207 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.756641    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:59:58.762745 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.762691    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:59:58.857087 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857044    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tls-certs\") pod \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") "
Nov 02 13:59:58.857294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857124    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-kube-api-access-bmh95\" (UniqueName: \"kubernetes.io/projected/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-kube-api-access-bmh95\") pod \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") "
Nov 02 13:59:58.857294 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857175    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-model-cache\") pod \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") "
Nov 02 13:59:58.857736 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857709    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-home\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-home\") pod \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") "
Nov 02 13:59:58.857834 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857774    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-model-cache\") pod \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") "
Nov 02 13:59:58.857898 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857861    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-dshm\") pod \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") "
Nov 02 13:59:58.857962 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857909    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-dshm\") pod \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") "
Nov 02 13:59:58.858023 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.857970    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tmp-dir\") pod \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") "
Nov 02 13:59:58.858080 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.858019    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tls-certs\") pod \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") "
Nov 02 13:59:58.858135 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.858103    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-home\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-home\") pod \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") "
Nov 02 13:59:58.858195 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.858166    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5fe786-8a48-4a93-9be5-e2092ee937a5-kube-api-access-bp5zs\" (UniqueName: \"kubernetes.io/projected/dc5fe786-8a48-4a93-9be5-e2092ee937a5-kube-api-access-bp5zs\") pod \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\" (UID: \"dc5fe786-8a48-4a93-9be5-e2092ee937a5\") "
Nov 02 13:59:58.858257 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.858194    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tmp-dir\") pod \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\" (UID: \"d3ff03bf-bb7f-4611-8a37-b9533e028dd8\") "
Nov 02 13:59:58.859238 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.859208    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tmp-dir" pod "d3ff03bf-bb7f-4611-8a37-b9533e028dd8" (UID: "d3ff03bf-bb7f-4611-8a37-b9533e028dd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.859439 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.859410    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-home" pod "dc5fe786-8a48-4a93-9be5-e2092ee937a5" (UID: "dc5fe786-8a48-4a93-9be5-e2092ee937a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.861260 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.861076    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-home" pod "d3ff03bf-bb7f-4611-8a37-b9533e028dd8" (UID: "d3ff03bf-bb7f-4611-8a37-b9533e028dd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.861746 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.861460    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-model-cache" pod "d3ff03bf-bb7f-4611-8a37-b9533e028dd8" (UID: "d3ff03bf-bb7f-4611-8a37-b9533e028dd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.862077 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.862056    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-model-cache" pod "dc5fe786-8a48-4a93-9be5-e2092ee937a5" (UID: "dc5fe786-8a48-4a93-9be5-e2092ee937a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.862398 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.862379    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tmp-dir" pod "dc5fe786-8a48-4a93-9be5-e2092ee937a5" (UID: "dc5fe786-8a48-4a93-9be5-e2092ee937a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.865685 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.865551    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-dshm" pod "dc5fe786-8a48-4a93-9be5-e2092ee937a5" (UID: "dc5fe786-8a48-4a93-9be5-e2092ee937a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.871639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.870112    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5fe786-8a48-4a93-9be5-e2092ee937a5-kube-api-access-bp5zs" pod "dc5fe786-8a48-4a93-9be5-e2092ee937a5" (UID: "dc5fe786-8a48-4a93-9be5-e2092ee937a5"). InnerVolumeSpecName "kube-api-access-bp5zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:58.871639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.870214    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tls-certs" pod "d3ff03bf-bb7f-4611-8a37-b9533e028dd8" (UID: "d3ff03bf-bb7f-4611-8a37-b9533e028dd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:58.871639 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.870347    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-dshm" pod "d3ff03bf-bb7f-4611-8a37-b9533e028dd8" (UID: "d3ff03bf-bb7f-4611-8a37-b9533e028dd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:58.871916 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.871895    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tls-certs" pod "dc5fe786-8a48-4a93-9be5-e2092ee937a5" (UID: "dc5fe786-8a48-4a93-9be5-e2092ee937a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:58.877794 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.877769    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-kube-api-access-bmh95" pod "d3ff03bf-bb7f-4611-8a37-b9533e028dd8" (UID: "d3ff03bf-bb7f-4611-8a37-b9533e028dd8"). InnerVolumeSpecName "kube-api-access-bmh95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:58.960527 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.960496    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.962870 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.960821    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5fe786-8a48-4a93-9be5-e2092ee937a5-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963046 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963031    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963198 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963186    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5zs\" (UniqueName: \"kubernetes.io/projected/dc5fe786-8a48-4a93-9be5-e2092ee937a5-kube-api-access-bp5zs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963279 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963266    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963359 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963349    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963432 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963423    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmh95\" (UniqueName: \"kubernetes.io/projected/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-kube-api-access-bmh95\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963507 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963497    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963626    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963792 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963780    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963862 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963853    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5fe786-8a48-4a93-9be5-e2092ee937a5-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.963937 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:58.963927    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ff03bf-bb7f-4611-8a37-b9533e028dd8-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 13:59:59.086589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.085681    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:59:59.086589 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.085740    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:59:59.116536 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.116454    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" podUID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:59:59.116972 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.116906    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" podUID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:59:59.178921 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.178889    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:59:59.179216 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.179188    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:59:59.323894 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.323847    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf" event={"ID":"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6","Type":"ContainerStarted","Data":"1e4611cd92d2901ca8b1b96b94e9249702242849af32266158b4a76135c74b8d"}
Nov 02 13:59:59.329953 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.329923    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf" event={"ID":"d3ff03bf-bb7f-4611-8a37-b9533e028dd8","Type":"ContainerDied","Data":"38818e402b556f70b30e9bd4b49d18928479fc4363885d2dc587942a8834f40b"}
Nov 02 13:59:59.330194 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.330176    5572 scope.go:122] "RemoveContainer" containerID="0ed31b4b7e74cf4e696d942d7b15088cfbe4516deac37b907e84b07accb435b8"
Nov 02 13:59:59.330657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.330637    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"
Nov 02 13:59:59.338696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.337744    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" event={"ID":"dc5fe786-8a48-4a93-9be5-e2092ee937a5","Type":"ContainerDied","Data":"b84a94acc72a07248c8f2000fe7c182c767c41c52369163c6f24ee2fafadc725"}
Nov 02 13:59:59.338696 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.337832    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"
Nov 02 13:59:59.386076 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.386003    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf" podStartSLOduration=2.385981052 podStartE2EDuration="2.385981052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:59.376330331 +0000 UTC m=+975.954419201" watchObservedRunningTime="2026-11-02 13:59:59.385981052 +0000 UTC m=+975.964069920"
Nov 02 13:59:59.402743 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.402704    5572 scope.go:122] "RemoveContainer" containerID="15399b7e8076dfdc465b07db73f51e324bc6d32e5be6478ab7443ca1b681a249"
Nov 02 13:59:59.440836 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.440806    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"]
Nov 02 13:59:59.477907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.477539    5572 patch_prober.go:29] interesting pod/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:59:59.477907 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.477630    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:59:59.482649 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.482616    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk"]
Nov 02 13:59:59.519440 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.519362    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"]
Nov 02 13:59:59.534996 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.534942    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f2e945/distinct-w1-f2e945-kserve-55b9fb9d5c-5wzzf"]
Nov 02 13:59:59.550773 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.550500    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" path="/var/lib/kubelet/pods/d3ff03bf-bb7f-4611-8a37-b9533e028dd8/volumes"
Nov 02 13:59:59.551657 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.551632    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" path="/var/lib/kubelet/pods/dc5fe786-8a48-4a93-9be5-e2092ee937a5/volumes"
Nov 02 13:59:59.706779 ip-10-0-128-62 kubenswrapper[5572]: I1102 13:59:59.705655    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f2e945/premium-w1-f2e945-kserve-55db7d458b-drszk" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 14:00:00.163171 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.163142    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"]
Nov 02 14:00:00.163863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.163844    5572 cpu_manager.go:414] "containerMap: removing container" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main"
Nov 02 14:00:00.163863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.163862    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main"
Nov 02 14:00:00.164026 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.163895    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main"
Nov 02 14:00:00.164026 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.163901    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main"
Nov 02 14:00:00.164026 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.164023    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5fe786-8a48-4a93-9be5-e2092ee937a5" containerName="main"
Nov 02 14:00:00.164175 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.164037    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ff03bf-bb7f-4611-8a37-b9533e028dd8" containerName="main"
Nov 02 14:00:00.170642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.170620    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.173835 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.173797    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"]
Nov 02 14:00:00.182259 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.182227    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.201427 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.201399    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"]
Nov 02 14:00:00.208210 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.208184    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.213151 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.213126    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"]
Nov 02 14:00:00.224918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.224886    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"]
Nov 02 14:00:00.231205 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.231180    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.232948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.232753    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"]
Nov 02 14:00:00.243743 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.243305    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"]
Nov 02 14:00:00.254663 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.254584    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"]
Nov 02 14:00:00.287256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.286737    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.287256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.286843    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb6d521-d893-4995-8331-a9b6dabf6a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.287256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.286894    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbnt\" (UniqueName: \"kubernetes.io/projected/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-kube-api-access-9vbnt\") pod \"maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.287256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.286974    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7156d74-04c2-4218-8782-4829ae754bfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-b4bqk\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.287256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.287031    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbxt\" (UniqueName: \"kubernetes.io/projected/3eb6d521-d893-4995-8331-a9b6dabf6a60-kube-api-access-9tbxt\") pod \"maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.287256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.287064    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72mf\" (UniqueName: \"kubernetes.io/projected/a7156d74-04c2-4218-8782-4829ae754bfa-kube-api-access-t72mf\") pod \"maas-api-key-cleanup-29893800-b4bqk\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.361753 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.360920    5572 generic.go:356] "Generic (PLEG): container finished" podID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerID="cb594dfef9f4b3f42467f53a68b97dea043d455554d854a72a83ce805c05801d" exitCode=0
Nov 02 14:00:00.362361 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.360990    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" event={"ID":"f82d4e2b-80d9-4195-827e-035f0b5f026a","Type":"ContainerDied","Data":"cb594dfef9f4b3f42467f53a68b97dea043d455554d854a72a83ce805c05801d"}
Nov 02 14:00:00.365448 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.365410    5572 generic.go:356] "Generic (PLEG): container finished" podID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerID="6b829521a0fa31b14a586d84abe04daabcef0ad5564b38d3fd9f8994f7dde93c" exitCode=0
Nov 02 14:00:00.365842 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.365632    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" event={"ID":"67594d7c-a1c4-4e31-bc0f-a635116017b1","Type":"ContainerDied","Data":"6b829521a0fa31b14a586d84abe04daabcef0ad5564b38d3fd9f8994f7dde93c"}
Nov 02 14:00:00.368629 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.368608    5572 generic.go:356] "Generic (PLEG): container finished" podID="0c210602-64e7-4655-b4f6-102c28acca45" containerID="af1904c4b7ce1bf0ea416493235f825ce3a7c3470720ae0f0a2ebe26a350192d" exitCode=0
Nov 02 14:00:00.368994 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.368959    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" event={"ID":"0c210602-64e7-4655-b4f6-102c28acca45","Type":"ContainerDied","Data":"af1904c4b7ce1bf0ea416493235f825ce3a7c3470720ae0f0a2ebe26a350192d"}
Nov 02 14:00:00.371762 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.371704    5572 generic.go:356] "Generic (PLEG): container finished" podID="d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" containerID="1e4611cd92d2901ca8b1b96b94e9249702242849af32266158b4a76135c74b8d" exitCode=0
Nov 02 14:00:00.372051 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.372027    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf" event={"ID":"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6","Type":"ContainerDied","Data":"1e4611cd92d2901ca8b1b96b94e9249702242849af32266158b4a76135c74b8d"}
Nov 02 14:00:00.388934 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.388896    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbnt\" (UniqueName: \"kubernetes.io/projected/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-kube-api-access-9vbnt\") pod \"maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.389199 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.389184    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7156d74-04c2-4218-8782-4829ae754bfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-b4bqk\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.389341 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.389327    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbxt\" (UniqueName: \"kubernetes.io/projected/3eb6d521-d893-4995-8331-a9b6dabf6a60-kube-api-access-9tbxt\") pod \"maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.389451 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.389437    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72mf\" (UniqueName: \"kubernetes.io/projected/a7156d74-04c2-4218-8782-4829ae754bfa-kube-api-access-t72mf\") pod \"maas-api-key-cleanup-29893800-b4bqk\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.389752 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.389718    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.389911 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.389896    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc76769-59a3-40d0-9b4e-bf5106d571e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.390089 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.390058    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzsb\" (UniqueName: \"kubernetes.io/projected/4bc76769-59a3-40d0-9b4e-bf5106d571e7-kube-api-access-cmzsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.390188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.390176    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb6d521-d893-4995-8331-a9b6dabf6a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.391241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.391218    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb6d521-d893-4995-8331-a9b6dabf6a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.391635 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.391615    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.391827 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.391807    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7156d74-04c2-4218-8782-4829ae754bfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-b4bqk\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.411027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.410898    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72mf\" (UniqueName: \"kubernetes.io/projected/a7156d74-04c2-4218-8782-4829ae754bfa-kube-api-access-t72mf\") pod \"maas-api-key-cleanup-29893800-b4bqk\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.411658 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.411632    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbxt\" (UniqueName: \"kubernetes.io/projected/3eb6d521-d893-4995-8331-a9b6dabf6a60-kube-api-access-9tbxt\") pod \"maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.414613 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.413238    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbnt\" (UniqueName: \"kubernetes.io/projected/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-kube-api-access-9vbnt\") pod \"maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.448394 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.448368    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 14:00:00.491912 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.491871    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc76769-59a3-40d0-9b4e-bf5106d571e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.492057 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.491969    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzsb\" (UniqueName: \"kubernetes.io/projected/4bc76769-59a3-40d0-9b4e-bf5106d571e7-kube-api-access-cmzsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.492650 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.492622    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc76769-59a3-40d0-9b4e-bf5106d571e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.509624 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.507666    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzsb\" (UniqueName: \"kubernetes.io/projected/4bc76769-59a3-40d0-9b4e-bf5106d571e7-kube-api-access-cmzsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.513027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.512967    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 14:00:00.552300 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.552273    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 14:00:00.593011 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.592971    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-credential-socket\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.593182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593021    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-credential-socket\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.593182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593058    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-socket\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.593182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593175    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-certs\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.593370 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593228    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-data\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.593370 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593268    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-credential-socket" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.593370 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593294    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-socket\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.593370 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593325    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-credential-socket" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.593370 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593346    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-token\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-token\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.593674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593404    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-kube-api-access-96jzp\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-kube-api-access-96jzp\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.593674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593457    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-data\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.593674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593444    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-socket" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.593674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593497    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-istio-token\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-istio-token\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.593674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593516    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-data" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.593674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593549    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-envoy\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.593674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593651    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-certs\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593614    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-socket" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593704    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-podinfo\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593728    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-data" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593740    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-kube-api-access-vwkwk\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-kube-api-access-vwkwk\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593796    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c210602-64e7-4655-b4f6-102c28acca45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c210602-64e7-4655-b4f6-102c28acca45-istio-podinfo\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593826    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67594d7c-a1c4-4e31-bc0f-a635116017b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67594d7c-a1c4-4e31-bc0f-a635116017b1-istiod-ca-cert\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.593884    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-envoy\") pod \"67594d7c-a1c4-4e31-bc0f-a635116017b1\" (UID: \"67594d7c-a1c4-4e31-bc0f-a635116017b1\") "
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.594022    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-certs" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.594548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.594110    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c210602-64e7-4655-b4f6-102c28acca45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c210602-64e7-4655-b4f6-102c28acca45-istiod-ca-cert\") pod \"0c210602-64e7-4655-b4f6-102c28acca45\" (UID: \"0c210602-64e7-4655-b4f6-102c28acca45\") "
Nov 02 14:00:00.595843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.595685    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.595843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.595708    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.595843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.595722    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.595843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.595758    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.595843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.595772    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.595843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.595785    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.595843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.595798    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.596311 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.596288    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c210602-64e7-4655-b4f6-102c28acca45-istiod-ca-cert" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:00.596551 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.596531    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-istio-token" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:00.598624 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.597992    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-token" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:00.598717 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.598649    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-envoy" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.601643 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.600932    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-kube-api-access-96jzp" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "kube-api-access-96jzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:00.601643 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.601469    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-envoy" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.601961 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.601875    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67594d7c-a1c4-4e31-bc0f-a635116017b1-istiod-ca-cert" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:00.602036 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.601994    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-kube-api-access-vwkwk" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "kube-api-access-vwkwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:00.602208 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.602184    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-podinfo" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:00.602500 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.602467    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c210602-64e7-4655-b4f6-102c28acca45-istio-podinfo" pod "0c210602-64e7-4655-b4f6-102c28acca45" (UID: "0c210602-64e7-4655-b4f6-102c28acca45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:00.603740 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.603713    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-certs" pod "67594d7c-a1c4-4e31-bc0f-a635116017b1" (UID: "67594d7c-a1c4-4e31-bc0f-a635116017b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.606787 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.606762    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:00.617222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.617201    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:00.639044 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.639018    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:00.659624 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.658969    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:00.696798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.696689    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-socket\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.696798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.696748    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82d4e2b-80d9-4195-827e-035f0b5f026a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82d4e2b-80d9-4195-827e-035f0b5f026a-istiod-ca-cert\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.696977 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.696800    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-podinfo\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.697032 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.697004    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-envoy\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.697085 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.697059    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-token\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-token\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.697169 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.697147    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-data\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.697247 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.697231    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-kube-api-access-5r95p\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-kube-api-access-5r95p\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.697308 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.697285    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-certs\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.697364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.697332    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-credential-socket\") pod \"f82d4e2b-80d9-4195-827e-035f0b5f026a\" (UID: \"f82d4e2b-80d9-4195-827e-035f0b5f026a\") "
Nov 02 14:00:00.698110 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698083    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698116    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwkwk\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-kube-api-access-vwkwk\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698131    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c210602-64e7-4655-b4f6-102c28acca45-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698144    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67594d7c-a1c4-4e31-bc0f-a635116017b1-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698157    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698168    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c210602-64e7-4655-b4f6-102c28acca45-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698182    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67594d7c-a1c4-4e31-bc0f-a635116017b1-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698196    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67594d7c-a1c4-4e31-bc0f-a635116017b1-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698209    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96jzp\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-kube-api-access-96jzp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698225    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c210602-64e7-4655-b4f6-102c28acca45-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.698666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.698237    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c210602-64e7-4655-b4f6-102c28acca45-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.699229 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.699200    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-data" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.699508 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.699484    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-credential-socket" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.699761 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.699741    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-certs" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.700030 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.700010    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82d4e2b-80d9-4195-827e-035f0b5f026a-istiod-ca-cert" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:00.700255 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.700231    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-socket" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.710016 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.709983    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-token" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:00.710016 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.709983    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-podinfo" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:00.718280 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.718229    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-kube-api-access-5r95p" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "kube-api-access-5r95p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:00.718969 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.718930    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-envoy" pod "f82d4e2b-80d9-4195-827e-035f0b5f026a" (UID: "f82d4e2b-80d9-4195-827e-035f0b5f026a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:00.800213 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800173    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800213 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800215    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800232    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800247    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r95p\" (UniqueName: \"kubernetes.io/projected/f82d4e2b-80d9-4195-827e-035f0b5f026a-kube-api-access-5r95p\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800266    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800279    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800292    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82d4e2b-80d9-4195-827e-035f0b5f026a-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800307    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82d4e2b-80d9-4195-827e-035f0b5f026a-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.800435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.800320    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82d4e2b-80d9-4195-827e-035f0b5f026a-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:00.984640 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:00.984613    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"]
Nov 02 14:00:01.002055 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:00:01.001234    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7156d74_04c2_4218_8782_4829ae754bfa.slice/crio-d2bb65593d4975a52f69c2c659403fe1dc2116846c8ecc6f7d3683df385640cc WatchSource:0}: Error finding container d2bb65593d4975a52f69c2c659403fe1dc2116846c8ecc6f7d3683df385640cc: Status 404 returned error can't find the container with id d2bb65593d4975a52f69c2c659403fe1dc2116846c8ecc6f7d3683df385640cc
Nov 02 14:00:01.005298 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.005264    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"]
Nov 02 14:00:01.009396 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:00:01.009226    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc76769_59a3_40d0_9b4e_bf5106d571e7.slice/crio-8eef3a7add78766617e40ad085ec03d575f6f9c2585af28b1b2c025a8d81e79b WatchSource:0}: Error finding container 8eef3a7add78766617e40ad085ec03d575f6f9c2585af28b1b2c025a8d81e79b: Status 404 returned error can't find the container with id 8eef3a7add78766617e40ad085ec03d575f6f9c2585af28b1b2c025a8d81e79b
Nov 02 14:00:01.013058 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.013034    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"]
Nov 02 14:00:01.021169 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.021137    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"]
Nov 02 14:00:01.385946 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.385907    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" event={"ID":"3eb6d521-d893-4995-8331-a9b6dabf6a60","Type":"ContainerStarted","Data":"0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5"}
Nov 02 14:00:01.386481 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.385974    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" event={"ID":"3eb6d521-d893-4995-8331-a9b6dabf6a60","Type":"ContainerStarted","Data":"ee59e949e3fef6fdf174c4e309fbc0a44acfe50325e39910272678b751a3cdc5"}
Nov 02 14:00:01.388165 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.388131    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" event={"ID":"4bc76769-59a3-40d0-9b4e-bf5106d571e7","Type":"ContainerStarted","Data":"8eef3a7add78766617e40ad085ec03d575f6f9c2585af28b1b2c025a8d81e79b"}
Nov 02 14:00:01.389735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.389715    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"
Nov 02 14:00:01.389735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.389724    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4" event={"ID":"f82d4e2b-80d9-4195-827e-035f0b5f026a","Type":"ContainerDied","Data":"d7737b265b3de56e50bf842f46c94a4848dad5d42b63b248b4caa0a442efc395"}
Nov 02 14:00:01.389908 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.389781    5572 scope.go:122] "RemoveContainer" containerID="cb594dfef9f4b3f42467f53a68b97dea043d455554d854a72a83ce805c05801d"
Nov 02 14:00:01.392696 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.392669    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" event={"ID":"0739d905-de97-4ba4-bbb8-85fdc3cf6e59","Type":"ContainerStarted","Data":"007096d83ca44e5c7e18f6a0cf94d9e83fe8c92578c628e94cc6c1107a4ab0c4"}
Nov 02 14:00:01.395689 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.395320    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"
Nov 02 14:00:01.395689 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.395363    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m" event={"ID":"67594d7c-a1c4-4e31-bc0f-a635116017b1","Type":"ContainerDied","Data":"227a5ae1b0f004f76ed03a368ff7fdaa872dfa6f9dbfa183e28beb2357734638"}
Nov 02 14:00:01.397222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.397198    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"
Nov 02 14:00:01.397306 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.397215    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb" event={"ID":"0c210602-64e7-4655-b4f6-102c28acca45","Type":"ContainerDied","Data":"6c759cfc1bb5036bf6df8dcb0258dc53fb0ae50c6ac079887f3d372ea8f9bf8b"}
Nov 02 14:00:01.398837 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.398813    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" event={"ID":"a7156d74-04c2-4218-8782-4829ae754bfa","Type":"ContainerStarted","Data":"d2bb65593d4975a52f69c2c659403fe1dc2116846c8ecc6f7d3683df385640cc"}
Nov 02 14:00:01.406808 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.406788    5572 scope.go:122] "RemoveContainer" containerID="6b829521a0fa31b14a586d84abe04daabcef0ad5564b38d3fd9f8994f7dde93c"
Nov 02 14:00:01.414135 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.414090    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" podStartSLOduration=1.41407563 podStartE2EDuration="1.41407563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:01.413128146 +0000 UTC m=+977.991217014" watchObservedRunningTime="2026-11-02 14:00:01.41407563 +0000 UTC m=+977.992164497"
Nov 02 14:00:01.420082 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.420054    5572 scope.go:122] "RemoveContainer" containerID="af1904c4b7ce1bf0ea416493235f825ce3a7c3470720ae0f0a2ebe26a350192d"
Nov 02 14:00:01.444750 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.444723    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"]
Nov 02 14:00:01.462658 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.462626    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-hgrm4"]
Nov 02 14:00:01.478356 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.478321    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"]
Nov 02 14:00:01.489954 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.486492    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-svx7m"]
Nov 02 14:00:01.499048 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.498868    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"]
Nov 02 14:00:01.505369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.505293    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d18fb-openshift-default-d9dddfb4c-ztjqb"]
Nov 02 14:00:01.554227 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.554143    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c210602-64e7-4655-b4f6-102c28acca45" path="/var/lib/kubelet/pods/0c210602-64e7-4655-b4f6-102c28acca45/volumes"
Nov 02 14:00:01.554687 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.554671    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" path="/var/lib/kubelet/pods/67594d7c-a1c4-4e31-bc0f-a635116017b1/volumes"
Nov 02 14:00:01.555115 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.555103    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" path="/var/lib/kubelet/pods/f82d4e2b-80d9-4195-827e-035f0b5f026a/volumes"
Nov 02 14:00:01.615543 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.615518    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 14:00:01.731680 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.731646    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-xks-ca\") pod \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") "
Nov 02 14:00:01.731859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.731826    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-openshift-ca\") pod \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") "
Nov 02 14:00:01.731915 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.731906    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-kube-api-access-xk9fj\" (UniqueName: \"kubernetes.io/projected/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-kube-api-access-xk9fj\") pod \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\" (UID: \"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6\") "
Nov 02 14:00:01.732196 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.732170    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-openshift-ca" pod "d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" (UID: "d2659dec-bd6d-4c65-af4c-dbd36bb46cd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:01.732414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.732398    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:01.733945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.733923    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-xks-ca" pod "d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" (UID: "d2659dec-bd6d-4c65-af4c-dbd36bb46cd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:01.734157 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.734134    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-kube-api-access-xk9fj" pod "d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" (UID: "d2659dec-bd6d-4c65-af4c-dbd36bb46cd6"). InnerVolumeSpecName "kube-api-access-xk9fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:01.834107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.834066    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9fj\" (UniqueName: \"kubernetes.io/projected/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-kube-api-access-xk9fj\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:01.834107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.834111    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:01.864012 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.863976    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"]
Nov 02 14:00:01.869664 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:01.869624    5572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb6d521_d893_4995_8331_a9b6dabf6a60.slice/crio-0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:00:01.879851 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:01.879822    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"]
Nov 02 14:00:02.238423 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.238386    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"]
Nov 02 14:00:02.249126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.249095    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1eb567-tn4vg"]
Nov 02 14:00:02.411618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.411507    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf" event={"ID":"d2659dec-bd6d-4c65-af4c-dbd36bb46cd6","Type":"ContainerDied","Data":"9edb62821d28b08fdeba2066b98a6820cc26e655bc4f8de07ba5554cb0d6df75"}
Nov 02 14:00:02.412010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.411520    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"
Nov 02 14:00:02.412010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.411611    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9edb62821d28b08fdeba2066b98a6820cc26e655bc4f8de07ba5554cb0d6df75"
Nov 02 14:00:02.413513 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.413485    5572 generic.go:356] "Generic (PLEG): container finished" podID="a7156d74-04c2-4218-8782-4829ae754bfa" containerID="8babaf98ac9efe9252457a7cd94a9791187108e661c3b19df15944fe44136c0c" exitCode=60
Nov 02 14:00:02.413667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.413605    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" event={"ID":"a7156d74-04c2-4218-8782-4829ae754bfa","Type":"ContainerDied","Data":"8babaf98ac9efe9252457a7cd94a9791187108e661c3b19df15944fe44136c0c"}
Nov 02 14:00:02.413899 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.413885    5572 scope.go:122] "RemoveContainer" containerID="8babaf98ac9efe9252457a7cd94a9791187108e661c3b19df15944fe44136c0c"
Nov 02 14:00:02.415396 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.415375    5572 generic.go:356] "Generic (PLEG): container finished" podID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerID="0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5" exitCode=6
Nov 02 14:00:02.415530 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.415515    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" event={"ID":"3eb6d521-d893-4995-8331-a9b6dabf6a60","Type":"ContainerDied","Data":"0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5"}
Nov 02 14:00:02.415790 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.415772    5572 scope.go:122] "RemoveContainer" containerID="0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5"
Nov 02 14:00:02.417754 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.417724    5572 generic.go:356] "Generic (PLEG): container finished" podID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerID="23ef879a04d027c3af0e9caecede2d8d0beb05e0bcc078c1a7fc4ae85408df2a" exitCode=60
Nov 02 14:00:02.418103 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.418086    5572 scope.go:122] "RemoveContainer" containerID="23ef879a04d027c3af0e9caecede2d8d0beb05e0bcc078c1a7fc4ae85408df2a"
Nov 02 14:00:02.418183 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.418125    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" event={"ID":"4bc76769-59a3-40d0-9b4e-bf5106d571e7","Type":"ContainerDied","Data":"23ef879a04d027c3af0e9caecede2d8d0beb05e0bcc078c1a7fc4ae85408df2a"}
Nov 02 14:00:02.422541 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.422514    5572 generic.go:356] "Generic (PLEG): container finished" podID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerID="40efa08a4830c54b7ee7955442edcd2bccb60ad735e505cbbb1ec958db2be595" exitCode=60
Nov 02 14:00:02.422704 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.422648    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" event={"ID":"0739d905-de97-4ba4-bbb8-85fdc3cf6e59","Type":"ContainerDied","Data":"40efa08a4830c54b7ee7955442edcd2bccb60ad735e505cbbb1ec958db2be595"}
Nov 02 14:00:02.422952 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:02.422935    5572 scope.go:122] "RemoveContainer" containerID="40efa08a4830c54b7ee7955442edcd2bccb60ad735e505cbbb1ec958db2be595"
Nov 02 14:00:03.142584 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.139098    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" podUID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:00:03.142584 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.139200    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"]
Nov 02 14:00:03.144743 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.144712    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f2e945-7475x"]
Nov 02 14:00:03.634251 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.581670    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerName="cleanup" containerID="cri-o://55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb" gracePeriod=30
Nov 02 14:00:03.660240 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.660073    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" podStartSLOduration=3.660049759 podStartE2EDuration="3.660049759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:03.642652884 +0000 UTC m=+980.220741754" watchObservedRunningTime="2026-11-02 14:00:03.660049759 +0000 UTC m=+980.238138625"
Nov 02 14:00:03.670780 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.670750    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd1839d-ea2a-4472-89e1-8e414e743dad" path="/var/lib/kubelet/pods/7bd1839d-ea2a-4472-89e1-8e414e743dad/volumes"
Nov 02 14:00:03.671459 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.671433    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ff0c33-61e8-4dee-8014-411f7b19389e" path="/var/lib/kubelet/pods/d9ff0c33-61e8-4dee-8014-411f7b19389e/volumes"
Nov 02 14:00:03.672015 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.671986    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" event={"ID":"a7156d74-04c2-4218-8782-4829ae754bfa","Type":"ContainerStarted","Data":"b7e2292979436c0a03904d71db5d90c1dad8d15c675d5b4dec0b2bb7e0211984"}
Nov 02 14:00:03.672115 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.672048    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" event={"ID":"3eb6d521-d893-4995-8331-a9b6dabf6a60","Type":"ContainerStarted","Data":"55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb"}
Nov 02 14:00:03.906697 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.906673    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:03.984433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.979963    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb6d521-d893-4995-8331-a9b6dabf6a60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb6d521-d893-4995-8331-a9b6dabf6a60-maas-api-service-ca\") pod \"3eb6d521-d893-4995-8331-a9b6dabf6a60\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") "
Nov 02 14:00:03.984433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.980016    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb6d521-d893-4995-8331-a9b6dabf6a60-kube-api-access-9tbxt\" (UniqueName: \"kubernetes.io/projected/3eb6d521-d893-4995-8331-a9b6dabf6a60-kube-api-access-9tbxt\") pod \"3eb6d521-d893-4995-8331-a9b6dabf6a60\" (UID: \"3eb6d521-d893-4995-8331-a9b6dabf6a60\") "
Nov 02 14:00:03.984433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:03.981478    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb6d521-d893-4995-8331-a9b6dabf6a60-maas-api-service-ca" pod "3eb6d521-d893-4995-8331-a9b6dabf6a60" (UID: "3eb6d521-d893-4995-8331-a9b6dabf6a60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:04.068719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.068657    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb6d521-d893-4995-8331-a9b6dabf6a60-kube-api-access-9tbxt" pod "3eb6d521-d893-4995-8331-a9b6dabf6a60" (UID: "3eb6d521-d893-4995-8331-a9b6dabf6a60"). InnerVolumeSpecName "kube-api-access-9tbxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:04.101673 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.100043    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb6d521-d893-4995-8331-a9b6dabf6a60-maas-api-service-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:04.101673 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.100079    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbxt\" (UniqueName: \"kubernetes.io/projected/3eb6d521-d893-4995-8331-a9b6dabf6a60-kube-api-access-9tbxt\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:04.172413 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.165652    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"]
Nov 02 14:00:04.233591 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.227494    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4fc49b-pkdmt"]
Nov 02 14:00:04.673183 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.673105    5572 generic.go:356] "Generic (PLEG): container finished" podID="a7156d74-04c2-4218-8782-4829ae754bfa" containerID="b7e2292979436c0a03904d71db5d90c1dad8d15c675d5b4dec0b2bb7e0211984" exitCode=60
Nov 02 14:00:04.673545 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.673186    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" event={"ID":"a7156d74-04c2-4218-8782-4829ae754bfa","Type":"ContainerDied","Data":"b7e2292979436c0a03904d71db5d90c1dad8d15c675d5b4dec0b2bb7e0211984"}
Nov 02 14:00:04.673545 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.673245    5572 scope.go:122] "RemoveContainer" containerID="8babaf98ac9efe9252457a7cd94a9791187108e661c3b19df15944fe44136c0c"
Nov 02 14:00:04.673545 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.673466    5572 scope.go:122] "RemoveContainer" containerID="b7e2292979436c0a03904d71db5d90c1dad8d15c675d5b4dec0b2bb7e0211984"
Nov 02 14:00:04.673863 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:04.673843    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-b4bqk_odh-ai-gateway-infra(a7156d74-04c2-4218-8782-4829ae754bfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" podUID="a7156d74-04c2-4218-8782-4829ae754bfa"
Nov 02 14:00:04.676123 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.676102    5572 generic.go:356] "Generic (PLEG): container finished" podID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerID="55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb" exitCode=6
Nov 02 14:00:04.676219 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.676154    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"
Nov 02 14:00:04.676219 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.676186    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" event={"ID":"3eb6d521-d893-4995-8331-a9b6dabf6a60","Type":"ContainerDied","Data":"55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb"}
Nov 02 14:00:04.676285 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.676229    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx" event={"ID":"3eb6d521-d893-4995-8331-a9b6dabf6a60","Type":"ContainerDied","Data":"ee59e949e3fef6fdf174c4e309fbc0a44acfe50325e39910272678b751a3cdc5"}
Nov 02 14:00:04.679794 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.678767    5572 generic.go:356] "Generic (PLEG): container finished" podID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerID="e194867c8cf2028bc8d00d107a539f1515598133e169ce8ac16b6c0cef1e46c9" exitCode=60
Nov 02 14:00:04.679794 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.678864    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" event={"ID":"4bc76769-59a3-40d0-9b4e-bf5106d571e7","Type":"ContainerDied","Data":"e194867c8cf2028bc8d00d107a539f1515598133e169ce8ac16b6c0cef1e46c9"}
Nov 02 14:00:04.679794 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.679341    5572 scope.go:122] "RemoveContainer" containerID="e194867c8cf2028bc8d00d107a539f1515598133e169ce8ac16b6c0cef1e46c9"
Nov 02 14:00:04.681735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.681711    5572 generic.go:356] "Generic (PLEG): container finished" podID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerID="359f78832fd1466058fdd3b400f969039518bf8dbbdc3e5fe33d2ca2edaecc8b" exitCode=60
Nov 02 14:00:04.681822 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.681788    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" event={"ID":"0739d905-de97-4ba4-bbb8-85fdc3cf6e59","Type":"ContainerDied","Data":"359f78832fd1466058fdd3b400f969039518bf8dbbdc3e5fe33d2ca2edaecc8b"}
Nov 02 14:00:04.682590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.682211    5572 scope.go:122] "RemoveContainer" containerID="359f78832fd1466058fdd3b400f969039518bf8dbbdc3e5fe33d2ca2edaecc8b"
Nov 02 14:00:04.682705 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:04.682645    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m_odh-ai-gateway-infra(4bc76769-59a3-40d0-9b4e-bf5106d571e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7"
Nov 02 14:00:04.682789 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:04.682705    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w_odh-ai-gateway-infra(0739d905-de97-4ba4-bbb8-85fdc3cf6e59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59"
Nov 02 14:00:04.704074 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.704053    5572 scope.go:122] "RemoveContainer" containerID="55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb"
Nov 02 14:00:04.717329 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.717307    5572 scope.go:122] "RemoveContainer" containerID="0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5"
Nov 02 14:00:04.729232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.729210    5572 scope.go:122] "RemoveContainer" containerID="55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb"
Nov 02 14:00:04.729579 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:04.729531    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb\": container with ID starting with 55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb not found: ID does not exist" containerID="55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb"
Nov 02 14:00:04.729671 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.729643    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb"} err="failed to get container status \"55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb\": rpc error: code = NotFound desc = could not find container \"55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb\": container with ID starting with 55b809a5a626d64a63a0859b3d4cc0631b4498f138c34f0329dbdf72a327d8cb not found: ID does not exist"
Nov 02 14:00:04.729671 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.729665    5572 scope.go:122] "RemoveContainer" containerID="0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5"
Nov 02 14:00:04.729955 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:04.729937    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5\": container with ID starting with 0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5 not found: ID does not exist" containerID="0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5"
Nov 02 14:00:04.730005 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.729961    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5"} err="failed to get container status \"0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5\": rpc error: code = NotFound desc = could not find container \"0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5\": container with ID starting with 0e3299372474b2989f2fd536eec6c403e5824d9fa68e751adb06f45b40820fa5 not found: ID does not exist"
Nov 02 14:00:04.730005 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.729975    5572 scope.go:122] "RemoveContainer" containerID="23ef879a04d027c3af0e9caecede2d8d0beb05e0bcc078c1a7fc4ae85408df2a"
Nov 02 14:00:04.752859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.745928    5572 scope.go:122] "RemoveContainer" containerID="40efa08a4830c54b7ee7955442edcd2bccb60ad735e505cbbb1ec958db2be595"
Nov 02 14:00:04.779652 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.779620    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"]
Nov 02 14:00:04.790524 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:04.790500    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1eb567-29893800-75qjx"]
Nov 02 14:00:05.544437 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:05.544404    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" path="/var/lib/kubelet/pods/3eb6d521-d893-4995-8331-a9b6dabf6a60/volumes"
Nov 02 14:00:05.545106 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:05.545077    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3aee62-baf3-4d58-82f0-33e5da695717" path="/var/lib/kubelet/pods/5f3aee62-baf3-4d58-82f0-33e5da695717/volumes"
Nov 02 14:00:05.692226 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:05.692202    5572 scope.go:122] "RemoveContainer" containerID="359f78832fd1466058fdd3b400f969039518bf8dbbdc3e5fe33d2ca2edaecc8b"
Nov 02 14:00:05.692697 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:05.692478    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w_odh-ai-gateway-infra(0739d905-de97-4ba4-bbb8-85fdc3cf6e59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59"
Nov 02 14:00:05.694085 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:05.694065    5572 scope.go:122] "RemoveContainer" containerID="b7e2292979436c0a03904d71db5d90c1dad8d15c675d5b4dec0b2bb7e0211984"
Nov 02 14:00:05.694345 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:05.694318    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-b4bqk_odh-ai-gateway-infra(a7156d74-04c2-4218-8782-4829ae754bfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" podUID="a7156d74-04c2-4218-8782-4829ae754bfa"
Nov 02 14:00:05.696915 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:05.696894    5572 scope.go:122] "RemoveContainer" containerID="e194867c8cf2028bc8d00d107a539f1515598133e169ce8ac16b6c0cef1e46c9"
Nov 02 14:00:05.697088 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:05.697071    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m_odh-ai-gateway-infra(4bc76769-59a3-40d0-9b4e-bf5106d571e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7"
Nov 02 14:00:06.348686 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.348663    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 14:00:06.365197 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.365176    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 14:00:06.491767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.491735    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-plugins-config-volume\") pod \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") "
Nov 02 14:00:06.491907 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.491772    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-kube-api-access-jn6xh\" (UniqueName: \"kubernetes.io/projected/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-kube-api-access-jn6xh\") pod \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\" (UID: \"e12c7cb2-83e6-49ef-824b-c75f61bbd77f\") "
Nov 02 14:00:06.491907 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.491842    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-kube-api-access-dp99b\" (UniqueName: \"kubernetes.io/projected/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-kube-api-access-dp99b\") pod \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") "
Nov 02 14:00:06.492157 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.492137    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-plugins-config-volume" pod "e12c7cb2-83e6-49ef-824b-c75f61bbd77f" (UID: "e12c7cb2-83e6-49ef-824b-c75f61bbd77f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:06.492203 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.492145    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-plugins-config-volume\") pod \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\" (UID: \"38391be5-0a0a-486d-b9a8-a6b9e4da6e93\") "
Nov 02 14:00:06.492481 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.492454    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-plugins-config-volume" pod "38391be5-0a0a-486d-b9a8-a6b9e4da6e93" (UID: "38391be5-0a0a-486d-b9a8-a6b9e4da6e93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:06.492884 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.492867    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:06.492982 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.492887    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:06.494059 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.494031    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-kube-api-access-dp99b" pod "38391be5-0a0a-486d-b9a8-a6b9e4da6e93" (UID: "38391be5-0a0a-486d-b9a8-a6b9e4da6e93"). InnerVolumeSpecName "kube-api-access-dp99b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:06.494131 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.494064    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-kube-api-access-jn6xh" pod "e12c7cb2-83e6-49ef-824b-c75f61bbd77f" (UID: "e12c7cb2-83e6-49ef-824b-c75f61bbd77f"). InnerVolumeSpecName "kube-api-access-jn6xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:06.594090 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.594025    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn6xh\" (UniqueName: \"kubernetes.io/projected/e12c7cb2-83e6-49ef-824b-c75f61bbd77f-kube-api-access-jn6xh\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:06.594090 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.594049    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp99b\" (UniqueName: \"kubernetes.io/projected/38391be5-0a0a-486d-b9a8-a6b9e4da6e93-kube-api-access-dp99b\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:06.706266 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.706236    5572 generic.go:356] "Generic (PLEG): container finished" podID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" containerID="526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38" exitCode=0
Nov 02 14:00:06.706718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.706282    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"
Nov 02 14:00:06.706718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.706325    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" event={"ID":"38391be5-0a0a-486d-b9a8-a6b9e4da6e93","Type":"ContainerDied","Data":"526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38"}
Nov 02 14:00:06.706718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.706377    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw" event={"ID":"38391be5-0a0a-486d-b9a8-a6b9e4da6e93","Type":"ContainerDied","Data":"977634c4d5cd52cc58d6a42c2e6dcd19df07c7f6fe4a14fa9df9d293c88e7124"}
Nov 02 14:00:06.706718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.706404    5572 scope.go:122] "RemoveContainer" containerID="526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38"
Nov 02 14:00:06.708241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.708218    5572 generic.go:356] "Generic (PLEG): container finished" podID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" containerID="78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4" exitCode=0
Nov 02 14:00:06.708365 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.708270    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"
Nov 02 14:00:06.708365 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.708339    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" event={"ID":"e12c7cb2-83e6-49ef-824b-c75f61bbd77f","Type":"ContainerDied","Data":"78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4"}
Nov 02 14:00:06.708454 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.708378    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf" event={"ID":"e12c7cb2-83e6-49ef-824b-c75f61bbd77f","Type":"ContainerDied","Data":"0591b8bb303f9082c4d103094022548b68e59a174b3a4db042e3df8528e9689b"}
Nov 02 14:00:06.719377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.719361    5572 scope.go:122] "RemoveContainer" containerID="526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38"
Nov 02 14:00:06.719656 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:06.719638    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38\": container with ID starting with 526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38 not found: ID does not exist" containerID="526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38"
Nov 02 14:00:06.719747 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.719662    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38"} err="failed to get container status \"526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38\": rpc error: code = NotFound desc = could not find container \"526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38\": container with ID starting with 526d5d324bba4ff44f68939404ee67a7497753c125f432d23125e3ae1d432b38 not found: ID does not exist"
Nov 02 14:00:06.719747 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.719677    5572 scope.go:122] "RemoveContainer" containerID="78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4"
Nov 02 14:00:06.730612 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.730593    5572 scope.go:122] "RemoveContainer" containerID="78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4"
Nov 02 14:00:06.730952 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:06.730897    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4\": container with ID starting with 78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4 not found: ID does not exist" containerID="78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4"
Nov 02 14:00:06.730952 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.730935    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4"} err="failed to get container status \"78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4\": rpc error: code = NotFound desc = could not find container \"78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4\": container with ID starting with 78c53680da6cecc6d9552f7b3156f63565d0c51c21f674f456734f796c97e0d4 not found: ID does not exist"
Nov 02 14:00:06.735438 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.735418    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"]
Nov 02 14:00:06.740348 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.740330    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4fc49b-645cfcc797-wkdtw"]
Nov 02 14:00:06.754149 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.754127    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"]
Nov 02 14:00:06.761292 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:06.761271    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4fc49b-5498477dc9-xk5qf"]
Nov 02 14:00:07.286907 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:07.286873    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"]
Nov 02 14:00:07.544213 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:07.544184    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" path="/var/lib/kubelet/pods/38391be5-0a0a-486d-b9a8-a6b9e4da6e93/volumes"
Nov 02 14:00:07.544666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:07.544647    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" path="/var/lib/kubelet/pods/e12c7cb2-83e6-49ef-824b-c75f61bbd77f/volumes"
Nov 02 14:00:07.545414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:07.545396    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"]
Nov 02 14:00:10.087101 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.087069    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"]
Nov 02 14:00:10.087960 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.087940    5572 cpu_manager.go:414] "containerMap: removing container" podUID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" containerName="payload-pre-processing"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.087963    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" containerName="payload-pre-processing"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.087978    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerName="cleanup"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.087985    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerName="cleanup"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088008    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" containerName="revoke-keys"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088015    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" containerName="revoke-keys"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088047    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088053    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088064    5572 cpu_manager.go:414] "containerMap: removing container" podUID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" containerName="payload-processing"
Nov 02 14:00:10.088083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088071    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" containerName="payload-processing"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088098    5572 cpu_manager.go:414] "containerMap: removing container" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088105    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088125    5572 cpu_manager.go:414] "containerMap: removing container" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088131    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088296    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="67594d7c-a1c4-4e31-bc0f-a635116017b1" containerName="istio-proxy"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088311    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82d4e2b-80d9-4195-827e-035f0b5f026a" containerName="istio-proxy"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088321    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerName="cleanup"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088335    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12c7cb2-83e6-49ef-824b-c75f61bbd77f" containerName="payload-pre-processing"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088345    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerName="cleanup"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088354    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" containerName="revoke-keys"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088366    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c210602-64e7-4655-b4f6-102c28acca45" containerName="istio-proxy"
Nov 02 14:00:10.088516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.088375    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="38391be5-0a0a-486d-b9a8-a6b9e4da6e93" containerName="payload-processing"
Nov 02 14:00:10.094377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.094355    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.102276 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.102250    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:10.112138 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.112117    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"]
Nov 02 14:00:10.176291 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.176250    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kck5g\" (UniqueName: \"kubernetes.io/projected/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-kube-api-access-kck5g\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.176481 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.176455    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.176648 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.176629    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.277837 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.277809    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.278008 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.277951    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kck5g\" (UniqueName: \"kubernetes.io/projected/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-kube-api-access-kck5g\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.278081 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.278035    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.278630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.278608    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.280331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.280309    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.293363 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.293339    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kck5g\" (UniqueName: \"kubernetes.io/projected/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-kube-api-access-kck5g\") pod \"maas-api-revoke-sub-54b6660c-kfbl9\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.417338 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.417261    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:10.722005 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.721934    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"]
Nov 02 14:00:10.814962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.813684    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"]
Nov 02 14:00:10.814962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.813957    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy" containerID="cri-o://97d38eb55cedafbc88770b89a03ac71334b8dd58521ff00b712816563ef786fd" gracePeriod=30
Nov 02 14:00:10.832686 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.832653    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"]
Nov 02 14:00:10.832901 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.832876    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" containerID="cri-o://f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887" gracePeriod=30
Nov 02 14:00:10.841078 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.837717    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"]
Nov 02 14:00:10.841078 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.838012    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy" containerID="cri-o://6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1" gracePeriod=30
Nov 02 14:00:10.845097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.845072    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"]
Nov 02 14:00:10.845269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.845248    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" containerID="cri-o://29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb" gracePeriod=30
Nov 02 14:00:10.855498 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:10.855471    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9" event={"ID":"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5","Type":"ContainerStarted","Data":"5ca3618c76ba9f873ef5007cba482d5bc58404a3955420445eec8512bd8478f2"}
Nov 02 14:00:11.928235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:11.928108    5572 generic.go:356] "Generic (PLEG): container finished" podID="f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" containerID="73f41a850dda6b3db4e7400d17728c4d45ecc8e25e28f8fb9ac267666470241d" exitCode=0
Nov 02 14:00:11.929305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:11.928252    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9" event={"ID":"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5","Type":"ContainerDied","Data":"73f41a850dda6b3db4e7400d17728c4d45ecc8e25e28f8fb9ac267666470241d"}
Nov 02 14:00:12.082391 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.082357    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"]
Nov 02 14:00:12.082663 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.082635    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" containerID="cri-o://bb718e678cab0068a58dc31e681339c857ff1ee566e2b93e7a259d53c958bd24" gracePeriod=30
Nov 02 14:00:12.104095 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.104066    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"]
Nov 02 14:00:12.104314 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.104275    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy" containerID="cri-o://02a7552ca953f9dddd09cc8bc18f31b78f567facb409e109b16c5adea230812e" gracePeriod=30
Nov 02 14:00:12.110206 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.110178    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"]
Nov 02 14:00:12.110533 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.110508    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy" containerID="cri-o://743505720ebb23a21abff1ef377277032c8689e6d3bc80d7aa0090d898b14603" gracePeriod=30
Nov 02 14:00:12.298648 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.295675    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" podUID="553121b1-c367-4a01-868c-e84169451981" containerName="payload-processing" containerID="cri-o://ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49" gracePeriod=25
Nov 02 14:00:12.898744 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.898709    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:00:12.898934 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:12.898775    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:00:13.136582 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.130716    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" podUID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:00:13.167642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.166020    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:13.237590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.228327    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-kube-api-access-kck5g\" (UniqueName: \"kubernetes.io/projected/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-kube-api-access-kck5g\") pod \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") "
Nov 02 14:00:13.237590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.228600    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-xks-ca\") pod \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") "
Nov 02 14:00:13.237590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.228705    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-openshift-ca\") pod \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\" (UID: \"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5\") "
Nov 02 14:00:13.237590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.229855    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-openshift-ca" pod "f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" (UID: "f778bc74-8a4e-4a4a-9321-594b2f6c3ca5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:13.254779 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.252693    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-xks-ca" pod "f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" (UID: "f778bc74-8a4e-4a4a-9321-594b2f6c3ca5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:13.273749 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.270258    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-kube-api-access-kck5g" pod "f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" (UID: "f778bc74-8a4e-4a4a-9321-594b2f6c3ca5"). InnerVolumeSpecName "kube-api-access-kck5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:13.342591 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.337930    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:13.342591 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.337969    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:13.342591 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.337986    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kck5g\" (UniqueName: \"kubernetes.io/projected/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5-kube-api-access-kck5g\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:13.488636 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:13.484802    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"]
Nov 02 14:00:14.089208 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.088669    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"
Nov 02 14:00:14.112585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.088536    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9" event={"ID":"f778bc74-8a4e-4a4a-9321-594b2f6c3ca5","Type":"ContainerDied","Data":"5ca3618c76ba9f873ef5007cba482d5bc58404a3955420445eec8512bd8478f2"}
Nov 02 14:00:14.112585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.099872    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ca3618c76ba9f873ef5007cba482d5bc58404a3955420445eec8512bd8478f2"
Nov 02 14:00:14.163586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.162661    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:00:14.163586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.162718    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:00:14.397644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.393935    5572 patch_prober.go:29] interesting pod/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:00:14.397644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.394013    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:00:14.811809 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.751963    5572 scope.go:122] "RemoveContainer" containerID="359f78832fd1466058fdd3b400f969039518bf8dbbdc3e5fe33d2ca2edaecc8b"
Nov 02 14:00:14.990705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:14.964912    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:15.199975 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.198010    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"]
Nov 02 14:00:15.199975 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.198342    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" containerID="cri-o://79c2b2385953ab955d7bd33146064a831c39aaa246eac1c09e1798336dfc06e9" gracePeriod=30
Nov 02 14:00:15.223657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.223625    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"]
Nov 02 14:00:15.224655 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.224621    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" containerID="cri-o://0af59780dcf0db223170c582010c2306fdbab2dbd3ff5ab001e5ab7e54f14bcf" gracePeriod=30
Nov 02 14:00:15.241863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.241772    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"]
Nov 02 14:00:15.242414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.242363    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy" containerID="cri-o://fb0350525175bab04b17f91ce61bd1c6b008d75b38305627db50ab2c1c4bcbb7" gracePeriod=30
Nov 02 14:00:15.277123 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.277091    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"]
Nov 02 14:00:15.278091 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.278011    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy" containerID="cri-o://586a3302b8cb024112b311c706d47fef1886c33185052d8c9b279d50efb14470" gracePeriod=30
Nov 02 14:00:15.322888 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.322541    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:00:15.322888 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.322648    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:00:15.460368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.460335    5572 patch_prober.go:29] interesting pod/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:00:15.460476 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.460396    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:00:15.477044 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.477020    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 14:00:15.502671 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.502648    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-plugins-config-volume\") pod \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") "
Nov 02 14:00:15.502772 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.502754    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-kube-api-access-rb47d\" (UniqueName: \"kubernetes.io/projected/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-kube-api-access-rb47d\") pod \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\" (UID: \"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0\") "
Nov 02 14:00:15.503050 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.503027    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-plugins-config-volume" pod "e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" (UID: "e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:15.503390 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.503368    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:15.505015 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.504987    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-kube-api-access-rb47d" pod "e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" (UID: "e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0"). InnerVolumeSpecName "kube-api-access-rb47d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:15.547402 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.547379    5572 scope.go:122] "RemoveContainer" containerID="e194867c8cf2028bc8d00d107a539f1515598133e169ce8ac16b6c0cef1e46c9"
Nov 02 14:00:15.605295 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:15.605267    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb47d\" (UniqueName: \"kubernetes.io/projected/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0-kube-api-access-rb47d\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.131781 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.131755    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 14:00:16.158027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.157980    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 14:00:16.164313 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.164284    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 14:00:16.212430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.212400    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-kube-api-access-q5grz\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-kube-api-access-q5grz\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.212896 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.212490    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-token\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-token\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.212896 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.212532    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-token\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-token\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.212896 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.212633    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-credential-socket\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.212896 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.212715    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-podinfo\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.213644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.213546    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-credential-socket" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.214177 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.214154    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-data" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.214618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.212765    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-data\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.214810 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.214789    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-socket" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.214885 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.214552    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-socket\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.214943 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.214879    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-socket\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.215350 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.215089    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-socket" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.215465 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.214911    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-podinfo\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.215650 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.215634    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-certs\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.216245 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.215989    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-envoy\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.216361 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.215925    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-certs" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.216361 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216299    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-token\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-token\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.216487 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216366    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-data\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.216700 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216680    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-certs\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.216786 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216744    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-credential-socket\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.216921 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216806    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45dfa51-e45e-4497-929c-4fbb9265331f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45dfa51-e45e-4497-929c-4fbb9265331f-istiod-ca-cert\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.216921 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216850    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-envoy\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.216921 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216885    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-podinfo\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.217078 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216921    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d021925-966b-4131-98b2-6b91d5ce6bc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d021925-966b-4131-98b2-6b91d5ce6bc3-istiod-ca-cert\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.217078 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.216948    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-socket\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.217078 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.217000    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-kube-api-access-cnnw9\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-kube-api-access-cnnw9\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.217078 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.217040    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-credential-socket\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.217078 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.217072    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-envoy\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.217329 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.217113    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-data\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.217329 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.217143    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-certs\") pod \"4d021925-966b-4131-98b2-6b91d5ce6bc3\" (UID: \"4d021925-966b-4131-98b2-6b91d5ce6bc3\") "
Nov 02 14:00:16.217329 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.217182    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe76bd4-674d-44a9-923f-2e473cb6e182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe76bd4-674d-44a9-923f-2e473cb6e182-istiod-ca-cert\") pod \"0fe76bd4-674d-44a9-923f-2e473cb6e182\" (UID: \"0fe76bd4-674d-44a9-923f-2e473cb6e182\") "
Nov 02 14:00:16.217329 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.217208    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-kube-api-access-vgwxs\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-kube-api-access-vgwxs\") pod \"d45dfa51-e45e-4497-929c-4fbb9265331f\" (UID: \"d45dfa51-e45e-4497-929c-4fbb9265331f\") "
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.218009    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d021925-966b-4131-98b2-6b91d5ce6bc3-istiod-ca-cert" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.218035    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.218057    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.218070    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.218083    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.218112    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.218829    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-podinfo" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.219403    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45dfa51-e45e-4497-929c-4fbb9265331f-istiod-ca-cert" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.219657    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-certs" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.219939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.219667    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-data" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.220720 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.220154    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-credential-socket" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.220720 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.220216    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-credential-socket" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.220823 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.220778    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-certs" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.221002 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.220956    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-data" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.221494 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.221250    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe76bd4-674d-44a9-923f-2e473cb6e182-istiod-ca-cert" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:16.221904 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.221469    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-socket" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.221904 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.221530    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-kube-api-access-vgwxs" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "kube-api-access-vgwxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.222749 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.222715    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-podinfo" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:16.224047 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.223893    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-envoy" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.224047 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.223946    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-token" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.224047 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.223978    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-token" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.224047 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.224008    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-kube-api-access-q5grz" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "kube-api-access-q5grz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.224047 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.224018    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-token" pod "d45dfa51-e45e-4497-929c-4fbb9265331f" (UID: "d45dfa51-e45e-4497-929c-4fbb9265331f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.224047 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.224041    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-podinfo" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:16.225464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.225440    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-envoy" pod "4d021925-966b-4131-98b2-6b91d5ce6bc3" (UID: "4d021925-966b-4131-98b2-6b91d5ce6bc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.225611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.225442    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-envoy" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.225611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.225526    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-kube-api-access-cnnw9" pod "0fe76bd4-674d-44a9-923f-2e473cb6e182" (UID: "0fe76bd4-674d-44a9-923f-2e473cb6e182"). InnerVolumeSpecName "kube-api-access-cnnw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.248827 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.248798    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" event={"ID":"4bc76769-59a3-40d0-9b4e-bf5106d571e7","Type":"ContainerStarted","Data":"53a9c9958fcbb89c68fb8abee160b5fb11e141f5d78d22fb5e0e6be90724704e"}
Nov 02 14:00:16.250896 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.250876    5572 generic.go:356] "Generic (PLEG): container finished" podID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerID="97d38eb55cedafbc88770b89a03ac71334b8dd58521ff00b712816563ef786fd" exitCode=0
Nov 02 14:00:16.250993 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.250949    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" event={"ID":"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad","Type":"ContainerDied","Data":"97d38eb55cedafbc88770b89a03ac71334b8dd58521ff00b712816563ef786fd"}
Nov 02 14:00:16.251055 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.251013    5572 scope.go:122] "RemoveContainer" containerID="1eed0bedaaf4a84bb719ff5e896b7619944e2c109e2e65c6a34d1e55a8896df1"
Nov 02 14:00:16.252448 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.252342    5572 generic.go:356] "Generic (PLEG): container finished" podID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerID="f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887" exitCode=0
Nov 02 14:00:16.252448 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.252371    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" event={"ID":"4d021925-966b-4131-98b2-6b91d5ce6bc3","Type":"ContainerDied","Data":"f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887"}
Nov 02 14:00:16.252448 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.252410    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"
Nov 02 14:00:16.252663 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.252414    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8" event={"ID":"4d021925-966b-4131-98b2-6b91d5ce6bc3","Type":"ContainerDied","Data":"2d95c70493a9388a3ca8da5f460bbb5c762a85a4eb07624b8a546a778fb38c7a"}
Nov 02 14:00:16.261771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.261739    5572 generic.go:356] "Generic (PLEG): container finished" podID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerID="9f4c927dd999b9df82a4bbb29fd4f97c1f24d9ac525b71430280ef361118bad8" exitCode=60
Nov 02 14:00:16.261888 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.261816    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" event={"ID":"0739d905-de97-4ba4-bbb8-85fdc3cf6e59","Type":"ContainerDied","Data":"9f4c927dd999b9df82a4bbb29fd4f97c1f24d9ac525b71430280ef361118bad8"}
Nov 02 14:00:16.262288 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.262266    5572 scope.go:122] "RemoveContainer" containerID="9f4c927dd999b9df82a4bbb29fd4f97c1f24d9ac525b71430280ef361118bad8"
Nov 02 14:00:16.262718 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:16.262690    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w_odh-ai-gateway-infra(0739d905-de97-4ba4-bbb8-85fdc3cf6e59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59"
Nov 02 14:00:16.263811 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.263708    5572 generic.go:356] "Generic (PLEG): container finished" podID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" containerID="e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8" exitCode=0
Nov 02 14:00:16.263914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.263854    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" event={"ID":"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0","Type":"ContainerDied","Data":"e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8"}
Nov 02 14:00:16.263914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.263902    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt" event={"ID":"e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0","Type":"ContainerDied","Data":"1bd3789a01e98349d6fc8fab9f6db49330e13e666bcc6459aec872bed16c353a"}
Nov 02 14:00:16.264031 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.263941    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"
Nov 02 14:00:16.268041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.266387    5572 generic.go:356] "Generic (PLEG): container finished" podID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerID="6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1" exitCode=0
Nov 02 14:00:16.268041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.266475    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" event={"ID":"0fe76bd4-674d-44a9-923f-2e473cb6e182","Type":"ContainerDied","Data":"6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1"}
Nov 02 14:00:16.268041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.266511    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct" event={"ID":"0fe76bd4-674d-44a9-923f-2e473cb6e182","Type":"ContainerDied","Data":"215e4c1f53d31870c5b6842c2be6bac19858809335ecff91764cc6e148cec839"}
Nov 02 14:00:16.268041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.266597    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.308376    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" podStartSLOduration=16.308355151 podStartE2EDuration="16.308355151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:16.274283228 +0000 UTC m=+992.852372110" watchObservedRunningTime="2026-11-02 14:00:16.308355151 +0000 UTC m=+992.886444021"
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.310287    5572 scope.go:122] "RemoveContainer" containerID="f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887"
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322436    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322465    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322540    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45dfa51-e45e-4497-929c-4fbb9265331f-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322574    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322588    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322599    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d021925-966b-4131-98b2-6b91d5ce6bc3-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322613    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322629    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnw9\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-kube-api-access-cnnw9\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322641    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322652    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322663    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322676    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d021925-966b-4131-98b2-6b91d5ce6bc3-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322688    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe76bd4-674d-44a9-923f-2e473cb6e182-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322699    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgwxs\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-kube-api-access-vgwxs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322711    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5grz\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-kube-api-access-q5grz\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322723    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322734    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322746    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322757    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322775    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45dfa51-e45e-4497-929c-4fbb9265331f-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322786    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d021925-966b-4131-98b2-6b91d5ce6bc3-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.323782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.322797    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe76bd4-674d-44a9-923f-2e473cb6e182-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.369010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.368463    5572 generic.go:356] "Generic (PLEG): container finished" podID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerID="29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb" exitCode=0
Nov 02 14:00:16.369010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.368519    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"
Nov 02 14:00:16.369010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.368686    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" event={"ID":"d45dfa51-e45e-4497-929c-4fbb9265331f","Type":"ContainerDied","Data":"29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb"}
Nov 02 14:00:16.369010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.368738    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq" event={"ID":"d45dfa51-e45e-4497-929c-4fbb9265331f","Type":"ContainerDied","Data":"ff4d923f1abfff09bb408848bf496bfa71ecc38434de3359d8e5330b129a9575"}
Nov 02 14:00:16.388235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.388208    5572 scope.go:122] "RemoveContainer" containerID="f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887"
Nov 02 14:00:16.388932 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:16.388905    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887\": container with ID starting with f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887 not found: ID does not exist" containerID="f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887"
Nov 02 14:00:16.389084 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.389059    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887"} err="failed to get container status \"f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887\": rpc error: code = NotFound desc = could not find container \"f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887\": container with ID starting with f7a6241936b09938ca8d507cc01fed5e9373895ddb66ec83b0b45b2cb3df2887 not found: ID does not exist"
Nov 02 14:00:16.389193 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.389182    5572 scope.go:122] "RemoveContainer" containerID="359f78832fd1466058fdd3b400f969039518bf8dbbdc3e5fe33d2ca2edaecc8b"
Nov 02 14:00:16.403852 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.400596    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"]
Nov 02 14:00:16.415549 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.415516    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-ljzl8"]
Nov 02 14:00:16.424850 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.424724    5572 scope.go:122] "RemoveContainer" containerID="e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8"
Nov 02 14:00:16.428683 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.428656    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"]
Nov 02 14:00:16.433942 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.433919    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f2e945-85cd4c6c47-722qt"]
Nov 02 14:00:16.440687 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.440662    5572 scope.go:122] "RemoveContainer" containerID="e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8"
Nov 02 14:00:16.441040 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:16.441019    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8\": container with ID starting with e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8 not found: ID does not exist" containerID="e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8"
Nov 02 14:00:16.441125 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.441054    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8"} err="failed to get container status \"e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8\": rpc error: code = NotFound desc = could not find container \"e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8\": container with ID starting with e0dcff2334a81b5489d8cb4862b478be7c913edbaf222aad7869debb38574be8 not found: ID does not exist"
Nov 02 14:00:16.441125 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.441078    5572 scope.go:122] "RemoveContainer" containerID="6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1"
Nov 02 14:00:16.441125 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.441081    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"]
Nov 02 14:00:16.446526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.446505    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-c9lct"]
Nov 02 14:00:16.455298 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.455274    5572 scope.go:122] "RemoveContainer" containerID="6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1"
Nov 02 14:00:16.455682 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:16.455656    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1\": container with ID starting with 6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1 not found: ID does not exist" containerID="6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1"
Nov 02 14:00:16.455744 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.455696    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1"} err="failed to get container status \"6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1\": rpc error: code = NotFound desc = could not find container \"6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1\": container with ID starting with 6cc508ff2574d27cd28d9ec3502e68196488f370cf94357d082f19f6515f99e1 not found: ID does not exist"
Nov 02 14:00:16.455744 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.455721    5572 scope.go:122] "RemoveContainer" containerID="29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb"
Nov 02 14:00:16.461105 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.461087    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"]
Nov 02 14:00:16.469647 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.469628    5572 scope.go:122] "RemoveContainer" containerID="29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb"
Nov 02 14:00:16.469989 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.469968    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-z84rq"]
Nov 02 14:00:16.470066 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:16.470037    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb\": container with ID starting with 29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb not found: ID does not exist" containerID="29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb"
Nov 02 14:00:16.470104 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.470064    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb"} err="failed to get container status \"29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb\": rpc error: code = NotFound desc = could not find container \"29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb\": container with ID starting with 29370662f83dc832e12e556c01c4452251cdccbadf790a438f83ebd119cf67eb not found: ID does not exist"
Nov 02 14:00:16.536471 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.536447    5572 scope.go:122] "RemoveContainer" containerID="b7e2292979436c0a03904d71db5d90c1dad8d15c675d5b4dec0b2bb7e0211984"
Nov 02 14:00:16.546153 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.546132    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 14:00:16.627176 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627111    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-podinfo\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627306 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627228    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-credential-socket\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627306 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627286    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-envoy\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627428 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627325    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-certs\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627428 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627388    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-kube-api-access-2bzpj\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-kube-api-access-2bzpj\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627428 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627411    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-data\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627601 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627480    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-token\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-token\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627601 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627523    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istiod-ca-cert\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627601 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627576    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-socket\") pod \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\" (UID: \"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad\") "
Nov 02 14:00:16.627769 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627595    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-credential-socket" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.627866 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627843    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-data" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.628368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627950    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istiod-ca-cert" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:16.628368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.627968    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-certs" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.628368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.628213    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-socket" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.628368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.628288    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.628368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.628309    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.628368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.628335    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.628368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.628349    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.681469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.681442    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-token" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.681469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.681450    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-envoy" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:16.681623 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.681468    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-podinfo" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:16.681623 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.681502    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-kube-api-access-2bzpj" pod "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" (UID: "0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad"). InnerVolumeSpecName "kube-api-access-2bzpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:16.729709 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.729682    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bzpj\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-kube-api-access-2bzpj\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.729709 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.729706    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.729846 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.729716    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.729846 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.729724    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.729846 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.729732    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:16.910407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:16.910377    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main" containerID="cri-o://e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe" gracePeriod=45
Nov 02 14:00:17.022294 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.022265    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"]
Nov 02 14:00:17.022976 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.022956    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy"
Nov 02 14:00:17.022976 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.022976    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.022989    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.022994    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023017    5572 cpu_manager.go:414] "containerMap: removing container" podUID="f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" containerName="revoke-keys"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023021    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" containerName="revoke-keys"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023030    5572 cpu_manager.go:414] "containerMap: removing container" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023034    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023048    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023052    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023070    5572 cpu_manager.go:414] "containerMap: removing container" podUID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" containerName="payload-pre-processing"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023074    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" containerName="payload-pre-processing"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023092    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023099    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023107    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerName="cleanup"
Nov 02 14:00:17.023126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023111    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb6d521-d893-4995-8331-a9b6dabf6a60" containerName="cleanup"
Nov 02 14:00:17.023518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023197    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" containerName="payload-pre-processing"
Nov 02 14:00:17.023518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023207    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" containerName="istio-proxy"
Nov 02 14:00:17.023518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023213    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy"
Nov 02 14:00:17.023518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023219    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" containerName="istio-proxy"
Nov 02 14:00:17.023518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023229    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" containerName="revoke-keys"
Nov 02 14:00:17.023518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023235    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" containerName="istio-proxy"
Nov 02 14:00:17.023518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.023241    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" containerName="istio-proxy"
Nov 02 14:00:17.031179 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.030882    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.038534 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.038507    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"]
Nov 02 14:00:17.040259 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.040239    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:17.135738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.135705    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2dc\" (UniqueName: \"kubernetes.io/projected/e78d60b0-75da-4133-b1f1-b878dbd7e292-kube-api-access-rv2dc\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.135923 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.135901    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.136041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.136025    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.238850 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.237550    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.238850 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.237707    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2dc\" (UniqueName: \"kubernetes.io/projected/e78d60b0-75da-4133-b1f1-b878dbd7e292-kube-api-access-rv2dc\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.238850 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.237873    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.238850 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.238803    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.244930 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.244908    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.271255 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.271218    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2dc\" (UniqueName: \"kubernetes.io/projected/e78d60b0-75da-4133-b1f1-b878dbd7e292-kube-api-access-rv2dc\") pod \"maas-api-revoke-sub-f607f6ad-5h85m\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.301450 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.301394    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 14:00:17.311220 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.311191    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"]
Nov 02 14:00:17.336251 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.336223    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"]
Nov 02 14:00:17.338734 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.338707    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12e3288-3bdd-4ca6-97cf-5d5af6062346-kube-api-access-5dr62\" (UniqueName: \"kubernetes.io/projected/c12e3288-3bdd-4ca6-97cf-5d5af6062346-kube-api-access-5dr62\") pod \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") "
Nov 02 14:00:17.338859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.338827    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-home\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-home\") pod \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") "
Nov 02 14:00:17.338925 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.338863    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tmp-dir\") pod \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") "
Nov 02 14:00:17.338966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.338940    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-dshm\") pod \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") "
Nov 02 14:00:17.339015 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.338993    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-model-cache\") pod \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") "
Nov 02 14:00:17.339067 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.339052    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tls-certs\") pod \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\" (UID: \"c12e3288-3bdd-4ca6-97cf-5d5af6062346\") "
Nov 02 14:00:17.340972 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.340246    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tmp-dir" pod "c12e3288-3bdd-4ca6-97cf-5d5af6062346" (UID: "c12e3288-3bdd-4ca6-97cf-5d5af6062346"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.340972 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.340745    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-model-cache" pod "c12e3288-3bdd-4ca6-97cf-5d5af6062346" (UID: "c12e3288-3bdd-4ca6-97cf-5d5af6062346"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.340972 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.340930    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-home" pod "c12e3288-3bdd-4ca6-97cf-5d5af6062346" (UID: "c12e3288-3bdd-4ca6-97cf-5d5af6062346"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.342465 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.342441    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tls-certs" pod "c12e3288-3bdd-4ca6-97cf-5d5af6062346" (UID: "c12e3288-3bdd-4ca6-97cf-5d5af6062346"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:17.346012 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.345976    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12e3288-3bdd-4ca6-97cf-5d5af6062346-kube-api-access-5dr62" pod "c12e3288-3bdd-4ca6-97cf-5d5af6062346" (UID: "c12e3288-3bdd-4ca6-97cf-5d5af6062346"). InnerVolumeSpecName "kube-api-access-5dr62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.349050 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.349023    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-dshm" pod "c12e3288-3bdd-4ca6-97cf-5d5af6062346" (UID: "c12e3288-3bdd-4ca6-97cf-5d5af6062346"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.350426 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.350204    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:17.386475 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.386451    5572 generic.go:356] "Generic (PLEG): container finished" podID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerID="bb718e678cab0068a58dc31e681339c857ff1ee566e2b93e7a259d53c958bd24" exitCode=0
Nov 02 14:00:17.386621 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.386520    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" event={"ID":"465e3231-6a1e-46f4-bdac-2e25cf860191","Type":"ContainerDied","Data":"bb718e678cab0068a58dc31e681339c857ff1ee566e2b93e7a259d53c958bd24"}
Nov 02 14:00:17.386621 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.386613    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h" event={"ID":"465e3231-6a1e-46f4-bdac-2e25cf860191","Type":"ContainerDied","Data":"503c8e1bffbd98c1e438e2d970144bdf30a692000d5b39b4e11de8ed1012016e"}
Nov 02 14:00:17.386735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.386640    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="503c8e1bffbd98c1e438e2d970144bdf30a692000d5b39b4e11de8ed1012016e"
Nov 02 14:00:17.389515 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.389476    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 14:00:17.390657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.389703    5572 generic.go:356] "Generic (PLEG): container finished" podID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerID="53a9c9958fcbb89c68fb8abee160b5fb11e141f5d78d22fb5e0e6be90724704e" exitCode=60
Nov 02 14:00:17.390657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.389803    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" event={"ID":"4bc76769-59a3-40d0-9b4e-bf5106d571e7","Type":"ContainerDied","Data":"53a9c9958fcbb89c68fb8abee160b5fb11e141f5d78d22fb5e0e6be90724704e"}
Nov 02 14:00:17.390657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.389852    5572 scope.go:122] "RemoveContainer" containerID="e194867c8cf2028bc8d00d107a539f1515598133e169ce8ac16b6c0cef1e46c9"
Nov 02 14:00:17.397027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.396976    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch" event={"ID":"0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad","Type":"ContainerDied","Data":"47965254e9299ab2752ac765ba556f12b02d7e46daa6db7fb7f1963779b7ea8c"}
Nov 02 14:00:17.397241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.397226    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"
Nov 02 14:00:17.405418 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.404971    5572 generic.go:356] "Generic (PLEG): container finished" podID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerID="02a7552ca953f9dddd09cc8bc18f31b78f567facb409e109b16c5adea230812e" exitCode=0
Nov 02 14:00:17.405418 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.405053    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" event={"ID":"dd26d6e3-9276-4d2a-8528-5ec79ac846cf","Type":"ContainerDied","Data":"02a7552ca953f9dddd09cc8bc18f31b78f567facb409e109b16c5adea230812e"}
Nov 02 14:00:17.409214 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.408686    5572 generic.go:356] "Generic (PLEG): container finished" podID="b772da3e-1119-4346-aa14-d376a1f137df" containerID="743505720ebb23a21abff1ef377277032c8689e6d3bc80d7aa0090d898b14603" exitCode=0
Nov 02 14:00:17.409214 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.408783    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" event={"ID":"b772da3e-1119-4346-aa14-d376a1f137df","Type":"ContainerDied","Data":"743505720ebb23a21abff1ef377277032c8689e6d3bc80d7aa0090d898b14603"}
Nov 02 14:00:17.433537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.415193    5572 generic.go:356] "Generic (PLEG): container finished" podID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerID="e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe" exitCode=0
Nov 02 14:00:17.433537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.415260    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"
Nov 02 14:00:17.433537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.415297    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" event={"ID":"c12e3288-3bdd-4ca6-97cf-5d5af6062346","Type":"ContainerDied","Data":"e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe"}
Nov 02 14:00:17.433537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.415329    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp" event={"ID":"c12e3288-3bdd-4ca6-97cf-5d5af6062346","Type":"ContainerDied","Data":"543d8429a6388e756c5ceaf39fbe346a2a3f4a0b7a6fcb72a4a8a85f2db628d0"}
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440312    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-certs\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440373    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-podinfo\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440406    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-token\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-token\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440540    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-data\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440620    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-socket\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440742    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-envoy\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440831    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-kube-api-access-2xgmf\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-kube-api-access-2xgmf\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440899    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-credential-socket\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.440963    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465e3231-6a1e-46f4-bdac-2e25cf860191-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465e3231-6a1e-46f4-bdac-2e25cf860191-istiod-ca-cert\") pod \"465e3231-6a1e-46f4-bdac-2e25cf860191\" (UID: \"465e3231-6a1e-46f4-bdac-2e25cf860191\") "
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.444607    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-data" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.445544 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.445426    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-certs" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.451617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.449162    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-token" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.451617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.449817    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-credential-socket" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.451617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.450346    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-socket" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.451617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.451527    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465e3231-6a1e-46f4-bdac-2e25cf860191-istiod-ca-cert" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:17.452918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.452894    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-podinfo" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:17.455208 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.455184    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-envoy" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.455685 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.455664    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-kube-api-access-2xgmf" pod "465e3231-6a1e-46f4-bdac-2e25cf860191" (UID: "465e3231-6a1e-46f4-bdac-2e25cf860191"). InnerVolumeSpecName "kube-api-access-2xgmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.458983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.458859    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.458983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.458890    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dr62\" (UniqueName: \"kubernetes.io/projected/c12e3288-3bdd-4ca6-97cf-5d5af6062346-kube-api-access-5dr62\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.458983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.458905    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.458983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.458919    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.458983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.458932    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.458983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.458944    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e3288-3bdd-4ca6-97cf-5d5af6062346-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.469129 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.468256    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" event={"ID":"a7156d74-04c2-4218-8782-4829ae754bfa","Type":"ContainerStarted","Data":"ca31d96387ed4875b7c068f8ab65d9510c0066fad4b402f6464ea1831df8b208"}
Nov 02 14:00:17.492920 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.488329    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 14:00:17.493256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.493198    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"]
Nov 02 14:00:17.501198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.493811    5572 scope.go:122] "RemoveContainer" containerID="97d38eb55cedafbc88770b89a03ac71334b8dd58521ff00b712816563ef786fd"
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.590912    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.590951    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xgmf\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-kube-api-access-2xgmf\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.590967    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.590980    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465e3231-6a1e-46f4-bdac-2e25cf860191-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.590994    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.591011    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.591022    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.591033    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.598235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.591047    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465e3231-6a1e-46f4-bdac-2e25cf860191-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.614866 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.614757    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 14:00:17.632869 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.632819    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe76bd4-674d-44a9-923f-2e473cb6e182" path="/var/lib/kubelet/pods/0fe76bd4-674d-44a9-923f-2e473cb6e182/volumes"
Nov 02 14:00:17.634314 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.633605    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d021925-966b-4131-98b2-6b91d5ce6bc3" path="/var/lib/kubelet/pods/4d021925-966b-4131-98b2-6b91d5ce6bc3/volumes"
Nov 02 14:00:17.635322 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.635280    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45dfa51-e45e-4497-929c-4fbb9265331f" path="/var/lib/kubelet/pods/d45dfa51-e45e-4497-929c-4fbb9265331f/volumes"
Nov 02 14:00:17.636377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.636360    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0" path="/var/lib/kubelet/pods/e5bf1d4a-09de-4b3b-a86f-d6a04d8f0bd0/volumes"
Nov 02 14:00:17.646864 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.646843    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1eb567-openshift-default-6c58cbc4b9-pbpch"]
Nov 02 14:00:17.656483 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.656457    5572 scope.go:122] "RemoveContainer" containerID="b013ec7780e1f9722b6c9e6c87e15b8d279b1612fae66e2bafde696c55926382"
Nov 02 14:00:17.682952 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.682924    5572 scope.go:122] "RemoveContainer" containerID="e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe"
Nov 02 14:00:17.697604 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.697534    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b772da3e-1119-4346-aa14-d376a1f137df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b772da3e-1119-4346-aa14-d376a1f137df-istio-podinfo\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.697732 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.697624    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-credential-socket\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.697732 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.697655    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-envoy\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.697860 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.697773    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-istio-token\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-istio-token\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.697860 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.697808    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-kube-api-access-tc5nj\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-kube-api-access-tc5nj\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.698618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.698049    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-socket\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.698618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.698138    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-certs\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.698618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.698168    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-data\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.698618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.698248    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b772da3e-1119-4346-aa14-d376a1f137df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b772da3e-1119-4346-aa14-d376a1f137df-istiod-ca-cert\") pod \"b772da3e-1119-4346-aa14-d376a1f137df\" (UID: \"b772da3e-1119-4346-aa14-d376a1f137df\") "
Nov 02 14:00:17.700950 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.700920    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-socket" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.701754 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.701159    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-credential-socket" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.708547 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.708527    5572 scope.go:122] "RemoveContainer" containerID="e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe"
Nov 02 14:00:17.709638 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.708863    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-data" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.709638 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.708903    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-certs" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.709638 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.708994    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b772da3e-1119-4346-aa14-d376a1f137df-istiod-ca-cert" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:17.709638 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:17.709271    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe\": container with ID starting with e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe not found: ID does not exist" containerID="e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe"
Nov 02 14:00:17.709638 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.709305    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe"} err="failed to get container status \"e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe\": rpc error: code = NotFound desc = could not find container \"e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe\": container with ID starting with e054d33a68bc912cfadebdd43b3f927539303225d81dbdc774e89de118b5e8fe not found: ID does not exist"
Nov 02 14:00:17.711835 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.711811    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b772da3e-1119-4346-aa14-d376a1f137df-istio-podinfo" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:17.712419 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.712397    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-kube-api-access-tc5nj" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "kube-api-access-tc5nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.713468 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.713446    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-istio-token" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.714619 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.714553    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-envoy" pod "b772da3e-1119-4346-aa14-d376a1f137df" (UID: "b772da3e-1119-4346-aa14-d376a1f137df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.753639 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.753611    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"]
Nov 02 14:00:17.758804 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.758760    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9e0c2f/distinct-2-w4-9e0c2f-kserve-7db67545b9-866fp"]
Nov 02 14:00:17.800449 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.800398    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-certs\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.800872 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.800849    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-kube-api-access-rwh5j\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-kube-api-access-rwh5j\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.801031 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.801016    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-envoy\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.801393 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.801365    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-socket\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.801590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.801548    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-podinfo\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.801773 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.801753    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-data\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.801910 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.801897    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-credential-socket\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.802077 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.802047    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istiod-ca-cert\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.802253 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.802237    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-token\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-token\") pod \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\" (UID: \"dd26d6e3-9276-4d2a-8528-5ec79ac846cf\") "
Nov 02 14:00:17.803433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.803407    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-data" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.803848 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.803826    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-certs" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.804075 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804048    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804079    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804094    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b772da3e-1119-4346-aa14-d376a1f137df-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804108    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804122    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b772da3e-1119-4346-aa14-d376a1f137df-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804136    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804149    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804162    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804175    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5nj\" (UniqueName: \"kubernetes.io/projected/b772da3e-1119-4346-aa14-d376a1f137df-kube-api-access-tc5nj\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804190    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804203    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b772da3e-1119-4346-aa14-d376a1f137df-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.804868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.804635    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-credential-socket" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.805340 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.805315    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istiod-ca-cert" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:17.805682 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.805658    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-socket" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.809388 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.809365    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-token" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.810761 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.810738    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-kube-api-access-rwh5j" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "kube-api-access-rwh5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.812074 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.812054    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-envoy" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:17.812290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.812267    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-podinfo" pod "dd26d6e3-9276-4d2a-8528-5ec79ac846cf" (UID: "dd26d6e3-9276-4d2a-8528-5ec79ac846cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:17.838611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.838184    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:17.840529 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.840506    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:17.905743 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.905714    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwh5j\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-kube-api-access-rwh5j\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.905743 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.905742    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.905936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.905752    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.905936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.905761    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.905936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.905771    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.905936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.905779    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.905936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.905786    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd26d6e3-9276-4d2a-8528-5ec79ac846cf-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:17.927160 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:00:17.927130    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78d60b0_75da_4133_b1f1_b878dbd7e292.slice/crio-34880c2f6f38d02e4019e9d2e0994f42385883189c35862a7e525bf20064c24e WatchSource:0}: Error finding container 34880c2f6f38d02e4019e9d2e0994f42385883189c35862a7e525bf20064c24e: Status 404 returned error can't find the container with id 34880c2f6f38d02e4019e9d2e0994f42385883189c35862a7e525bf20064c24e
Nov 02 14:00:17.927994 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:17.927970    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"]
Nov 02 14:00:18.006946 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.006913    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-kube-api-access-9vbnt\" (UniqueName: \"kubernetes.io/projected/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-kube-api-access-9vbnt\") pod \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") "
Nov 02 14:00:18.007070 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.006962    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc76769-59a3-40d0-9b4e-bf5106d571e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc76769-59a3-40d0-9b4e-bf5106d571e7-maas-api-service-ca\") pod \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") "
Nov 02 14:00:18.007070 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.006996    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-maas-api-service-ca\") pod \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\" (UID: \"0739d905-de97-4ba4-bbb8-85fdc3cf6e59\") "
Nov 02 14:00:18.007070 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.007029    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc76769-59a3-40d0-9b4e-bf5106d571e7-kube-api-access-cmzsb\" (UniqueName: \"kubernetes.io/projected/4bc76769-59a3-40d0-9b4e-bf5106d571e7-kube-api-access-cmzsb\") pod \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\" (UID: \"4bc76769-59a3-40d0-9b4e-bf5106d571e7\") "
Nov 02 14:00:18.007364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.007337    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc76769-59a3-40d0-9b4e-bf5106d571e7-maas-api-service-ca" pod "4bc76769-59a3-40d0-9b4e-bf5106d571e7" (UID: "4bc76769-59a3-40d0-9b4e-bf5106d571e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:18.007364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.007333    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-maas-api-service-ca" pod "0739d905-de97-4ba4-bbb8-85fdc3cf6e59" (UID: "0739d905-de97-4ba4-bbb8-85fdc3cf6e59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:18.007536 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.007489    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc76769-59a3-40d0-9b4e-bf5106d571e7-maas-api-service-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:18.007536 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.007511    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-maas-api-service-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:18.009207 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.009179    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc76769-59a3-40d0-9b4e-bf5106d571e7-kube-api-access-cmzsb" pod "4bc76769-59a3-40d0-9b4e-bf5106d571e7" (UID: "4bc76769-59a3-40d0-9b4e-bf5106d571e7"). InnerVolumeSpecName "kube-api-access-cmzsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:18.009298 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.009217    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-kube-api-access-9vbnt" pod "0739d905-de97-4ba4-bbb8-85fdc3cf6e59" (UID: "0739d905-de97-4ba4-bbb8-85fdc3cf6e59"). InnerVolumeSpecName "kube-api-access-9vbnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:18.041782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.041758    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" podUID="553121b1-c367-4a01-868c-e84169451981" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:00:18.108437 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.108413    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vbnt\" (UniqueName: \"kubernetes.io/projected/0739d905-de97-4ba4-bbb8-85fdc3cf6e59-kube-api-access-9vbnt\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:18.108437 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.108438    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzsb\" (UniqueName: \"kubernetes.io/projected/4bc76769-59a3-40d0-9b4e-bf5106d571e7-kube-api-access-cmzsb\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:18.478940 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.478910    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"
Nov 02 14:00:18.479403 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.478914    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m" event={"ID":"4bc76769-59a3-40d0-9b4e-bf5106d571e7","Type":"ContainerDied","Data":"8eef3a7add78766617e40ad085ec03d575f6f9c2585af28b1b2c025a8d81e79b"}
Nov 02 14:00:18.479403 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.479029    5572 scope.go:122] "RemoveContainer" containerID="53a9c9958fcbb89c68fb8abee160b5fb11e141f5d78d22fb5e0e6be90724704e"
Nov 02 14:00:18.481866 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.481845    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"
Nov 02 14:00:18.481995 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.481857    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w" event={"ID":"0739d905-de97-4ba4-bbb8-85fdc3cf6e59","Type":"ContainerDied","Data":"007096d83ca44e5c7e18f6a0cf94d9e83fe8c92578c628e94cc6c1107a4ab0c4"}
Nov 02 14:00:18.483377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.483361    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"
Nov 02 14:00:18.483377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.483377    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5" event={"ID":"dd26d6e3-9276-4d2a-8528-5ec79ac846cf","Type":"ContainerDied","Data":"497f0847d9e0ecc0adc272f6c294cd6051d3edd8e9421e85933d58f59beee301"}
Nov 02 14:00:18.485221 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.485116    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" podUID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" containerName="payload-processing" containerID="cri-o://a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81" gracePeriod=25
Nov 02 14:00:18.487590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.485844    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m" event={"ID":"e78d60b0-75da-4133-b1f1-b878dbd7e292","Type":"ContainerStarted","Data":"517dd8b0ff53c1e00ff1eae660a65dd6ddbff0998b530bf8e967109b9ad3e287"}
Nov 02 14:00:18.487590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.485889    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m" event={"ID":"e78d60b0-75da-4133-b1f1-b878dbd7e292","Type":"ContainerStarted","Data":"34880c2f6f38d02e4019e9d2e0994f42385883189c35862a7e525bf20064c24e"}
Nov 02 14:00:18.488614 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.488497    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9" event={"ID":"b772da3e-1119-4346-aa14-d376a1f137df","Type":"ContainerDied","Data":"d18175e740addadc582c6870c55ac4435dfc87111cfe37d588ebcd46b1a3f40a"}
Nov 02 14:00:18.488703 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.488646    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"
Nov 02 14:00:18.494682 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.494653    5572 generic.go:356] "Generic (PLEG): container finished" podID="a7156d74-04c2-4218-8782-4829ae754bfa" containerID="ca31d96387ed4875b7c068f8ab65d9510c0066fad4b402f6464ea1831df8b208" exitCode=60
Nov 02 14:00:18.494799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.494775    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" event={"ID":"a7156d74-04c2-4218-8782-4829ae754bfa","Type":"ContainerDied","Data":"ca31d96387ed4875b7c068f8ab65d9510c0066fad4b402f6464ea1831df8b208"}
Nov 02 14:00:18.494959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.494928    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"
Nov 02 14:00:18.495396 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.495031    5572 scope.go:122] "RemoveContainer" containerID="9f4c927dd999b9df82a4bbb29fd4f97c1f24d9ac525b71430280ef361118bad8"
Nov 02 14:00:18.495396 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.495260    5572 scope.go:122] "RemoveContainer" containerID="ca31d96387ed4875b7c068f8ab65d9510c0066fad4b402f6464ea1831df8b208"
Nov 02 14:00:18.495921 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:18.495509    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-b4bqk_odh-ai-gateway-infra(a7156d74-04c2-4218-8782-4829ae754bfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" podUID="a7156d74-04c2-4218-8782-4829ae754bfa"
Nov 02 14:00:18.505756 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.505741    5572 scope.go:122] "RemoveContainer" containerID="02a7552ca953f9dddd09cc8bc18f31b78f567facb409e109b16c5adea230812e"
Nov 02 14:00:18.515700 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.515680    5572 scope.go:122] "RemoveContainer" containerID="743505720ebb23a21abff1ef377277032c8689e6d3bc80d7aa0090d898b14603"
Nov 02 14:00:18.526799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.526766    5572 scope.go:122] "RemoveContainer" containerID="b7e2292979436c0a03904d71db5d90c1dad8d15c675d5b4dec0b2bb7e0211984"
Nov 02 14:00:18.582872 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.582796    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m" podStartSLOduration=2.582775866 podStartE2EDuration="2.582775866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:18.548359796 +0000 UTC m=+995.126448665" watchObservedRunningTime="2026-11-02 14:00:18.582775866 +0000 UTC m=+995.160864735"
Nov 02 14:00:18.602583 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.597831    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"]
Nov 02 14:00:18.606988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.606421    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-28p8h"]
Nov 02 14:00:18.626842 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.625085    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"]
Nov 02 14:00:18.632180 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.632153    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a9443-29893800-bpv6m"]
Nov 02 14:00:18.637488 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.637466    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"]
Nov 02 14:00:18.651872 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.651843    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-54bs5"]
Nov 02 14:00:18.677124 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.677094    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"]
Nov 02 14:00:18.691164 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.691140    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9e0c2f-29893800-hrq9w"]
Nov 02 14:00:18.704602 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.704570    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"]
Nov 02 14:00:18.710472 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.710453    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"]
Nov 02 14:00:18.721360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:18.721338    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f2e945-openshift-default-6b59fbb98f-8qkq9"]
Nov 02 14:00:19.514889 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.514855    5572 generic.go:356] "Generic (PLEG): container finished" podID="e78d60b0-75da-4133-b1f1-b878dbd7e292" containerID="517dd8b0ff53c1e00ff1eae660a65dd6ddbff0998b530bf8e967109b9ad3e287" exitCode=0
Nov 02 14:00:19.515342 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.515005    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m" event={"ID":"e78d60b0-75da-4133-b1f1-b878dbd7e292","Type":"ContainerDied","Data":"517dd8b0ff53c1e00ff1eae660a65dd6ddbff0998b530bf8e967109b9ad3e287"}
Nov 02 14:00:19.546888 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.546859    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" path="/var/lib/kubelet/pods/0739d905-de97-4ba4-bbb8-85fdc3cf6e59/volumes"
Nov 02 14:00:19.547390 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.547376    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad" path="/var/lib/kubelet/pods/0b79d4c7-4fb0-4647-823a-e6a4f1cd13ad/volumes"
Nov 02 14:00:19.547855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.547841    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" path="/var/lib/kubelet/pods/465e3231-6a1e-46f4-bdac-2e25cf860191/volumes"
Nov 02 14:00:19.548619 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.548603    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" path="/var/lib/kubelet/pods/4bc76769-59a3-40d0-9b4e-bf5106d571e7/volumes"
Nov 02 14:00:19.549033 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.549020    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b772da3e-1119-4346-aa14-d376a1f137df" path="/var/lib/kubelet/pods/b772da3e-1119-4346-aa14-d376a1f137df/volumes"
Nov 02 14:00:19.549398 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.549386    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" path="/var/lib/kubelet/pods/c12e3288-3bdd-4ca6-97cf-5d5af6062346/volumes"
Nov 02 14:00:19.550017 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.549999    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" path="/var/lib/kubelet/pods/dd26d6e3-9276-4d2a-8528-5ec79ac846cf/volumes"
Nov 02 14:00:19.772179 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.772117    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:19.832036 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.832008    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7156d74-04c2-4218-8782-4829ae754bfa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7156d74-04c2-4218-8782-4829ae754bfa-maas-api-service-ca\") pod \"a7156d74-04c2-4218-8782-4829ae754bfa\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") "
Nov 02 14:00:19.832197 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.832104    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7156d74-04c2-4218-8782-4829ae754bfa-kube-api-access-t72mf\" (UniqueName: \"kubernetes.io/projected/a7156d74-04c2-4218-8782-4829ae754bfa-kube-api-access-t72mf\") pod \"a7156d74-04c2-4218-8782-4829ae754bfa\" (UID: \"a7156d74-04c2-4218-8782-4829ae754bfa\") "
Nov 02 14:00:19.832404 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.832380    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7156d74-04c2-4218-8782-4829ae754bfa-maas-api-service-ca" pod "a7156d74-04c2-4218-8782-4829ae754bfa" (UID: "a7156d74-04c2-4218-8782-4829ae754bfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:19.832655 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.832637    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7156d74-04c2-4218-8782-4829ae754bfa-maas-api-service-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:19.834290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.834269    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7156d74-04c2-4218-8782-4829ae754bfa-kube-api-access-t72mf" pod "a7156d74-04c2-4218-8782-4829ae754bfa" (UID: "a7156d74-04c2-4218-8782-4829ae754bfa"). InnerVolumeSpecName "kube-api-access-t72mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:19.934326 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:19.934297    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t72mf\" (UniqueName: \"kubernetes.io/projected/a7156d74-04c2-4218-8782-4829ae754bfa-kube-api-access-t72mf\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.386618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.373095    5572 patch_prober.go:29] interesting pod/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 14:00:20.386618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.373153    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 14:00:20.543716 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.541041    5572 generic.go:356] "Generic (PLEG): container finished" podID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerID="586a3302b8cb024112b311c706d47fef1886c33185052d8c9b279d50efb14470" exitCode=0
Nov 02 14:00:20.543716 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.541127    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" event={"ID":"8a5c27fc-880e-498d-bd52-7b1d208f2de4","Type":"ContainerDied","Data":"586a3302b8cb024112b311c706d47fef1886c33185052d8c9b279d50efb14470"}
Nov 02 14:00:20.543716 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.543601    5572 generic.go:356] "Generic (PLEG): container finished" podID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerID="fb0350525175bab04b17f91ce61bd1c6b008d75b38305627db50ab2c1c4bcbb7" exitCode=0
Nov 02 14:00:20.544295 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.543953    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" event={"ID":"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5","Type":"ContainerDied","Data":"fb0350525175bab04b17f91ce61bd1c6b008d75b38305627db50ab2c1c4bcbb7"}
Nov 02 14:00:20.548395 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.548368    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk" event={"ID":"a7156d74-04c2-4218-8782-4829ae754bfa","Type":"ContainerDied","Data":"d2bb65593d4975a52f69c2c659403fe1dc2116846c8ecc6f7d3683df385640cc"}
Nov 02 14:00:20.548677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.548421    5572 scope.go:122] "RemoveContainer" containerID="ca31d96387ed4875b7c068f8ab65d9510c0066fad4b402f6464ea1831df8b208"
Nov 02 14:00:20.548677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.548546    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"
Nov 02 14:00:20.558023 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.556321    5572 generic.go:356] "Generic (PLEG): container finished" podID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerID="0af59780dcf0db223170c582010c2306fdbab2dbd3ff5ab001e5ab7e54f14bcf" exitCode=0
Nov 02 14:00:20.558023 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.556411    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" event={"ID":"60c36986-d034-4084-a190-9a6aad2c8ef0","Type":"ContainerDied","Data":"0af59780dcf0db223170c582010c2306fdbab2dbd3ff5ab001e5ab7e54f14bcf"}
Nov 02 14:00:20.560122 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.560071    5572 generic.go:356] "Generic (PLEG): container finished" podID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerID="79c2b2385953ab955d7bd33146064a831c39aaa246eac1c09e1798336dfc06e9" exitCode=0
Nov 02 14:00:20.560382 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.560361    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" event={"ID":"3eeee583-d5dc-42eb-84a0-a5aa32d80498","Type":"ContainerDied","Data":"79c2b2385953ab955d7bd33146064a831c39aaa246eac1c09e1798336dfc06e9"}
Nov 02 14:00:20.567597 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.565902    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 14:00:20.633800 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.632203    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"]
Nov 02 14:00:20.647684 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.643497    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 14:00:20.649173 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.649135    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-b4bqk"]
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662004    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-kube-api-access-8d2qd\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-kube-api-access-8d2qd\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662041    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-envoy\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662077    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-credential-socket\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662109    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istiod-ca-cert\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662155    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-kube-api-access-89f2v\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-kube-api-access-89f2v\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662190    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-credential-socket\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662212    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-socket\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662264    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-token\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-token\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662289    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-envoy\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662313    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-podinfo\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662340    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-token\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-token\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662364    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c36986-d034-4084-a190-9a6aad2c8ef0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c36986-d034-4084-a190-9a6aad2c8ef0-istiod-ca-cert\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662389    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-socket\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662438    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-podinfo\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662480    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-certs\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662500    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-data\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.662931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.662553    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-certs\") pod \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\" (UID: \"8a5c27fc-880e-498d-bd52-7b1d208f2de4\") "
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.678812    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-data\") pod \"60c36986-d034-4084-a190-9a6aad2c8ef0\" (UID: \"60c36986-d034-4084-a190-9a6aad2c8ef0\") "
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.663140    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-credential-socket" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.681058    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-data" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.681820    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-kube-api-access-8d2qd" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "kube-api-access-8d2qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.681746    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-token" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.682073    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-credential-socket" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.682276    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-socket" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.682623    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c36986-d034-4084-a190-9a6aad2c8ef0-istiod-ca-cert" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:20.683053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.682693    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istiod-ca-cert" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:20.690626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.683821    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-certs" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.690626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.684002    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-socket" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.690626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.684231    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-data" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.690626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.684418    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-certs" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.690626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.685131    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-kube-api-access-89f2v" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "kube-api-access-89f2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.690626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.687517    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-token" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.695204 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.695170    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-podinfo" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:20.697355 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.697315    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-envoy" pod "60c36986-d034-4084-a190-9a6aad2c8ef0" (UID: "60c36986-d034-4084-a190-9a6aad2c8ef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.697820 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.697799    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-podinfo" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:20.704477 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.703831    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-envoy" pod "8a5c27fc-880e-498d-bd52-7b1d208f2de4" (UID: "8a5c27fc-880e-498d-bd52-7b1d208f2de4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.711967 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.711945    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 14:00:20.738099 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.738070    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781243    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-data\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781295    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-certs\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781327    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-podinfo\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781371    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istiod-ca-cert\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781422    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-credential-socket\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781453    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-podinfo\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781497    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-credential-socket\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781593    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-data\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781633    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-envoy\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781703    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-kube-api-access-r4jzz\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-kube-api-access-r4jzz\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.781798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.781734    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-token\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-token\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.782906 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.782839    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-kube-api-access-478gx\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-kube-api-access-478gx\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.782906 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.782889    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-certs\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.783006 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.782919    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-token\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-token\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.783006 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.782961    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-socket\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.783006 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.782991    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-socket\") pod \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\" (UID: \"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5\") "
Nov 02 14:00:20.783206 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783057    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-envoy\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.783206 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783085    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istiod-ca-cert\") pod \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\" (UID: \"3eeee583-d5dc-42eb-84a0-a5aa32d80498\") "
Nov 02 14:00:20.783504 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783486    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783507    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c36986-d034-4084-a190-9a6aad2c8ef0-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783525    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783538    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783551    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783601    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783615    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783628    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783642    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2qd\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-kube-api-access-8d2qd\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783657    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783672    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783687    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783701    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89f2v\" (UniqueName: \"kubernetes.io/projected/60c36986-d034-4084-a190-9a6aad2c8ef0-kube-api-access-89f2v\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783714    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783728    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c36986-d034-4084-a190-9a6aad2c8ef0-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783741    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783755    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5c27fc-880e-498d-bd52-7b1d208f2de4-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.783988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.783768    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c36986-d034-4084-a190-9a6aad2c8ef0-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.784447 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.784014    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-socket" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.784447 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.784103    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istiod-ca-cert" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:20.784447 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.784236    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-certs" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.784666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.784444    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-data" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.784928 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.784898    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-socket" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.785393 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.785370    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-certs" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.785941 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.785917    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istiod-ca-cert" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:20.788199 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.787654    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-podinfo" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:20.788199 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.787859    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-credential-socket" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.788199 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.788021    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-credential-socket" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.788199 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.788170    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-data" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.789416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.789380    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-token" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.789661 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.789628    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-envoy" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.791600 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.791527    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-podinfo" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:20.791696 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.791637    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-envoy" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:20.796178 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.796118    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-kube-api-access-478gx" pod "3eeee583-d5dc-42eb-84a0-a5aa32d80498" (UID: "3eeee583-d5dc-42eb-84a0-a5aa32d80498"). InnerVolumeSpecName "kube-api-access-478gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.796848 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.796827    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-token" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.797531 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.797497    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-kube-api-access-r4jzz" pod "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" (UID: "9e4a77d6-bd5b-435d-b57c-fe6cea575bd5"). InnerVolumeSpecName "kube-api-access-r4jzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.876583 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.870891    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892518    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-478gx\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-kube-api-access-478gx\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892585    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892602    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892616    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892632    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892646    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892659    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892673    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892750    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892765    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892778    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892795    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eeee583-d5dc-42eb-84a0-a5aa32d80498-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892808    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eeee583-d5dc-42eb-84a0-a5aa32d80498-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892823    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892836    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892848    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892862    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4jzz\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-kube-api-access-r4jzz\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.896324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.892876    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.993248 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.993221    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78d60b0-75da-4133-b1f1-b878dbd7e292-kube-api-access-rv2dc\" (UniqueName: \"kubernetes.io/projected/e78d60b0-75da-4133-b1f1-b878dbd7e292-kube-api-access-rv2dc\") pod \"e78d60b0-75da-4133-b1f1-b878dbd7e292\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") "
Nov 02 14:00:20.993407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.993293    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-xks-ca\") pod \"e78d60b0-75da-4133-b1f1-b878dbd7e292\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") "
Nov 02 14:00:20.993506 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.993483    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-openshift-ca\") pod \"e78d60b0-75da-4133-b1f1-b878dbd7e292\" (UID: \"e78d60b0-75da-4133-b1f1-b878dbd7e292\") "
Nov 02 14:00:20.993888 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.993863    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-openshift-ca" pod "e78d60b0-75da-4133-b1f1-b878dbd7e292" (UID: "e78d60b0-75da-4133-b1f1-b878dbd7e292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:20.994217 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.994197    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.995514 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.995493    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-xks-ca" pod "e78d60b0-75da-4133-b1f1-b878dbd7e292" (UID: "e78d60b0-75da-4133-b1f1-b878dbd7e292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:20.995592 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:20.995513    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78d60b0-75da-4133-b1f1-b878dbd7e292-kube-api-access-rv2dc" pod "e78d60b0-75da-4133-b1f1-b878dbd7e292" (UID: "e78d60b0-75da-4133-b1f1-b878dbd7e292"). InnerVolumeSpecName "kube-api-access-rv2dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:21.095860 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.095831    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv2dc\" (UniqueName: \"kubernetes.io/projected/e78d60b0-75da-4133-b1f1-b878dbd7e292-kube-api-access-rv2dc\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:21.095860 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.095859    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78d60b0-75da-4133-b1f1-b878dbd7e292-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:21.432391 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:21.432294    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:00:21.543147 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.543116    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" path="/var/lib/kubelet/pods/a7156d74-04c2-4218-8782-4829ae754bfa/volumes"
Nov 02 14:00:21.570249 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.570228    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"
Nov 02 14:00:21.570609 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.570259    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m" event={"ID":"e78d60b0-75da-4133-b1f1-b878dbd7e292","Type":"ContainerDied","Data":"34880c2f6f38d02e4019e9d2e0994f42385883189c35862a7e525bf20064c24e"}
Nov 02 14:00:21.570609 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.570299    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34880c2f6f38d02e4019e9d2e0994f42385883189c35862a7e525bf20064c24e"
Nov 02 14:00:21.572984 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.572965    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"
Nov 02 14:00:21.572984 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.572975    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd" event={"ID":"60c36986-d034-4084-a190-9a6aad2c8ef0","Type":"ContainerDied","Data":"6e6fcbfffdc22d394240bf5da657fa445f1d56ae6e592672982bec7e8b45660b"}
Nov 02 14:00:21.573149 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.573026    5572 scope.go:122] "RemoveContainer" containerID="0af59780dcf0db223170c582010c2306fdbab2dbd3ff5ab001e5ab7e54f14bcf"
Nov 02 14:00:21.574623 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.574605    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"
Nov 02 14:00:21.574912 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.574889    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz" event={"ID":"3eeee583-d5dc-42eb-84a0-a5aa32d80498","Type":"ContainerDied","Data":"cc7a1e033912c80c51c62464dd6c98305709db2108a3ae2aa13e49682575a8bd"}
Nov 02 14:00:21.576456 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.576434    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g" event={"ID":"8a5c27fc-880e-498d-bd52-7b1d208f2de4","Type":"ContainerDied","Data":"7121f33343caf5ea6bb84833e2acd98b71455c02497dd24724f744909de4e79d"}
Nov 02 14:00:21.576554 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.576529    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"
Nov 02 14:00:21.578327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.578292    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg" event={"ID":"9e4a77d6-bd5b-435d-b57c-fe6cea575bd5","Type":"ContainerDied","Data":"15c17c3f9dca60c5a5cfa2fe6d7b924555338fa2ade45afc849f477ea8fddf02"}
Nov 02 14:00:21.578413 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.578385    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"
Nov 02 14:00:21.585796 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.585780    5572 scope.go:122] "RemoveContainer" containerID="79c2b2385953ab955d7bd33146064a831c39aaa246eac1c09e1798336dfc06e9"
Nov 02 14:00:21.597094 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.597076    5572 scope.go:122] "RemoveContainer" containerID="586a3302b8cb024112b311c706d47fef1886c33185052d8c9b279d50efb14470"
Nov 02 14:00:21.604352 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.604318    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"]
Nov 02 14:00:21.608703 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.608681    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hzvjd"]
Nov 02 14:00:21.608952 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.608941    5572 scope.go:122] "RemoveContainer" containerID="fb0350525175bab04b17f91ce61bd1c6b008d75b38305627db50ab2c1c4bcbb7"
Nov 02 14:00:21.619970 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.619952    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"]
Nov 02 14:00:21.624858 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.624840    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-hj2dz"]
Nov 02 14:00:21.640109 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.640090    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"]
Nov 02 14:00:21.645963 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.645942    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-n5v8g"]
Nov 02 14:00:21.654294 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.654275    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"]
Nov 02 14:00:21.660060 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:21.660040    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4fc49b-openshift-default-798b7b5cc5-jflvg"]
Nov 02 14:00:22.075690 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:22.075656    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" podUID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:00:22.568019 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:22.567979    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" containerID="cri-o://fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd" gracePeriod=45
Nov 02 14:00:22.906115 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:22.906089    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 14:00:23.034167 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034088    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-home\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-home\") pod \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") "
Nov 02 14:00:23.034310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034162    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tmp-dir\") pod \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") "
Nov 02 14:00:23.034310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034200    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b3588b-c669-447e-9bd6-f07c5a8298d0-kube-api-access-bxjm5\" (UniqueName: \"kubernetes.io/projected/d7b3588b-c669-447e-9bd6-f07c5a8298d0-kube-api-access-bxjm5\") pod \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") "
Nov 02 14:00:23.034310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034245    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tls-certs\") pod \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") "
Nov 02 14:00:23.034310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034292    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-dshm\") pod \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") "
Nov 02 14:00:23.034528 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034386    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-home" pod "d7b3588b-c669-447e-9bd6-f07c5a8298d0" (UID: "d7b3588b-c669-447e-9bd6-f07c5a8298d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:23.034528 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034396    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tmp-dir" pod "d7b3588b-c669-447e-9bd6-f07c5a8298d0" (UID: "d7b3588b-c669-447e-9bd6-f07c5a8298d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:23.034528 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034443    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-model-cache\") pod \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\" (UID: \"d7b3588b-c669-447e-9bd6-f07c5a8298d0\") "
Nov 02 14:00:23.034822 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.034797    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-model-cache" pod "d7b3588b-c669-447e-9bd6-f07c5a8298d0" (UID: "d7b3588b-c669-447e-9bd6-f07c5a8298d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:23.035071 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.035055    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:23.035123 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.035073    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:23.035123 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.035081    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:23.036881 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.036851    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-dshm" pod "d7b3588b-c669-447e-9bd6-f07c5a8298d0" (UID: "d7b3588b-c669-447e-9bd6-f07c5a8298d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:23.037040 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.037019    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tls-certs" pod "d7b3588b-c669-447e-9bd6-f07c5a8298d0" (UID: "d7b3588b-c669-447e-9bd6-f07c5a8298d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:23.037106 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.037053    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b3588b-c669-447e-9bd6-f07c5a8298d0-kube-api-access-bxjm5" pod "d7b3588b-c669-447e-9bd6-f07c5a8298d0" (UID: "d7b3588b-c669-447e-9bd6-f07c5a8298d0"). InnerVolumeSpecName "kube-api-access-bxjm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:23.136728 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.136653    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxjm5\" (UniqueName: \"kubernetes.io/projected/d7b3588b-c669-447e-9bd6-f07c5a8298d0-kube-api-access-bxjm5\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:23.136728 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.136683    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b3588b-c669-447e-9bd6-f07c5a8298d0-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:23.136728 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.136694    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b3588b-c669-447e-9bd6-f07c5a8298d0-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:23.587919 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.587882    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" path="/var/lib/kubelet/pods/3eeee583-d5dc-42eb-84a0-a5aa32d80498/volumes"
Nov 02 14:00:23.590412 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.588608    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" path="/var/lib/kubelet/pods/60c36986-d034-4084-a190-9a6aad2c8ef0/volumes"
Nov 02 14:00:23.590412 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.589234    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" path="/var/lib/kubelet/pods/8a5c27fc-880e-498d-bd52-7b1d208f2de4/volumes"
Nov 02 14:00:23.590412 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.590402    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" path="/var/lib/kubelet/pods/9e4a77d6-bd5b-435d-b57c-fe6cea575bd5/volumes"
Nov 02 14:00:23.611723 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.611695    5572 generic.go:356] "Generic (PLEG): container finished" podID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerID="fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd" exitCode=0
Nov 02 14:00:23.611844 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.611745    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"
Nov 02 14:00:23.611844 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.611776    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" event={"ID":"d7b3588b-c669-447e-9bd6-f07c5a8298d0","Type":"ContainerDied","Data":"fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd"}
Nov 02 14:00:23.611844 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.611825    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" event={"ID":"d7b3588b-c669-447e-9bd6-f07c5a8298d0","Type":"ContainerDied","Data":"4b45f9dd9838292404f74280b2b47708fe9e325797d098ed2f450072a6b8ba42"}
Nov 02 14:00:23.611992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.611846    5572 scope.go:122] "RemoveContainer" containerID="fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd"
Nov 02 14:00:23.623255 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.623234    5572 scope.go:122] "RemoveContainer" containerID="fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd"
Nov 02 14:00:23.623597 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:23.623553    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd\": container with ID starting with fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd not found: ID does not exist" containerID="fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd"
Nov 02 14:00:23.623688 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.623610    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd"} err="failed to get container status \"fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd\": rpc error: code = NotFound desc = could not find container \"fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd\": container with ID starting with fb09fee4207cc206f194f82dd02a0ad71438ad14d0e43f00fa5c3f2226233ecd not found: ID does not exist"
Nov 02 14:00:23.639720 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.639700    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"]
Nov 02 14:00:23.646287 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.646268    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd"]
Nov 02 14:00:23.878590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:23.878480    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7a9443/simulator-w0-7a9443-kserve-6656cf55c9-5l2vd" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:25.445763 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:25.445736    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"]
Nov 02 14:00:25.453207 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:25.453183    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9e0c2f-wpvzf"]
Nov 02 14:00:25.542339 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:25.542311    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2659dec-bd6d-4c65-af4c-dbd36bb46cd6" path="/var/lib/kubelet/pods/d2659dec-bd6d-4c65-af4c-dbd36bb46cd6/volumes"
Nov 02 14:00:25.542756 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:25.542742    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" path="/var/lib/kubelet/pods/d7b3588b-c669-447e-9bd6-f07c5a8298d0/volumes"
Nov 02 14:00:28.041588 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.041495    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" podUID="553121b1-c367-4a01-868c-e84169451981" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:00:28.410584 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.390971    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"]
Nov 02 14:00:28.411589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.411539    5572 cpu_manager.go:414] "containerMap: removing container" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy"
Nov 02 14:00:28.411750 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.411736    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy"
Nov 02 14:00:28.411843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.411833    5572 cpu_manager.go:414] "containerMap: removing container" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy"
Nov 02 14:00:28.411918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.411908    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy"
Nov 02 14:00:28.412008 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.411998    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy"
Nov 02 14:00:28.412084 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412075    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy"
Nov 02 14:00:28.412158 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412149    5572 cpu_manager.go:414] "containerMap: removing container" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy"
Nov 02 14:00:28.412230 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412219    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy"
Nov 02 14:00:28.412309 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412300    5572 cpu_manager.go:414] "containerMap: removing container" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy"
Nov 02 14:00:28.412379 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412370    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy"
Nov 02 14:00:28.412463 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412455    5572 cpu_manager.go:414] "containerMap: removing container" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy"
Nov 02 14:00:28.412536 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412528    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy"
Nov 02 14:00:28.412640 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412629    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.412717 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412707    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.412808 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412799    5572 cpu_manager.go:414] "containerMap: removing container" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy"
Nov 02 14:00:28.412876 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412868    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy"
Nov 02 14:00:28.412956 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.412947    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.413025 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413016    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.413106 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413097    5572 cpu_manager.go:414] "containerMap: removing container" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:28.413176 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413166    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:28.413248 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413239    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.413315 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413305    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.413403 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413393    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main"
Nov 02 14:00:28.413482 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413471    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main"
Nov 02 14:00:28.413580 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.413551    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.466554    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470652    5572 cpu_manager.go:414] "containerMap: removing container" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470662    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470715    5572 cpu_manager.go:414] "containerMap: removing container" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470726    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470747    5572 cpu_manager.go:414] "containerMap: removing container" podUID="e78d60b0-75da-4133-b1f1-b878dbd7e292" containerName="revoke-keys"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470756    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78d60b0-75da-4133-b1f1-b878dbd7e292" containerName="revoke-keys"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470768    5572 cpu_manager.go:414] "containerMap: removing container" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470776    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470787    5572 cpu_manager.go:414] "containerMap: removing container" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470795    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470806    5572 cpu_manager.go:414] "containerMap: removing container" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470813    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470982    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78d60b0-75da-4133-b1f1-b878dbd7e292" containerName="revoke-keys"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.470999    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471010    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4a77d6-bd5b-435d-b57c-fe6cea575bd5" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471018    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471027    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471036    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c36986-d034-4084-a190-9a6aad2c8ef0" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471045    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471055    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eeee583-d5dc-42eb-84a0-a5aa32d80498" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471065    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5c27fc-880e-498d-bd52-7b1d208f2de4" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471075    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd26d6e3-9276-4d2a-8528-5ec79ac846cf" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471084    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="465e3231-6a1e-46f4-bdac-2e25cf860191" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471092    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471104    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b772da3e-1119-4346-aa14-d376a1f137df" containerName="istio-proxy"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471112    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471123    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0739d905-de97-4ba4-bbb8-85fdc3cf6e59" containerName="cleanup"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471132    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12e3288-3bdd-4ca6-97cf-5d5af6062346" containerName="main"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471140    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b3588b-c669-447e-9bd6-f07c5a8298d0" containerName="main"
Nov 02 14:00:28.503593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.471147    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:28.541546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.511895    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.541546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.513041    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"]
Nov 02 14:00:28.541546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.523713    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:28.541546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.523807    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:00:28.697589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.692330    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dv9m\" (UniqueName: \"kubernetes.io/projected/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-kube-api-access-5dv9m\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.697589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.692596    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.697589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.692687    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.795315 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.795269    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.796594 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.795611    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.796594 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.795664    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dv9m\" (UniqueName: \"kubernetes.io/projected/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-kube-api-access-5dv9m\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.797442 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.797419    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.816080 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.816054    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:28.834666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:28.834639    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dv9m\" (UniqueName: \"kubernetes.io/projected/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-kube-api-access-5dv9m\") pod \"maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:29.046835 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:29.046745    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:29.572847 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:29.572825    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"]
Nov 02 14:00:29.788307 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:29.788265    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk" event={"ID":"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa","Type":"ContainerStarted","Data":"faee591fdd54e500c4a423cc260ad3a2e259b0f8e9ee292a906e11acf1759e57"}
Nov 02 14:00:30.798664 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:30.798553    5572 generic.go:356] "Generic (PLEG): container finished" podID="1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" containerID="6cf740e3cdcd555c560a78a0bfe4f06b0263aadc161581c4de67434f2b4ef66b" exitCode=0
Nov 02 14:00:30.798664 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:30.798587    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk" event={"ID":"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa","Type":"ContainerDied","Data":"6cf740e3cdcd555c560a78a0bfe4f06b0263aadc161581c4de67434f2b4ef66b"}
Nov 02 14:00:31.968355 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:31.968327    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:31.997163 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:31.997128    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-kube-api-access-5dv9m\" (UniqueName: \"kubernetes.io/projected/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-kube-api-access-5dv9m\") pod \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") "
Nov 02 14:00:31.997310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:31.997187    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-openshift-ca\") pod \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") "
Nov 02 14:00:31.997310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:31.997223    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-xks-ca\") pod \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\" (UID: \"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa\") "
Nov 02 14:00:31.998302 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:31.998273    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-openshift-ca" pod "1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" (UID: "1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:32.000696 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.000669    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-xks-ca" pod "1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" (UID: "1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:32.002179 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.002156    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-kube-api-access-5dv9m" pod "1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" (UID: "1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa"). InnerVolumeSpecName "kube-api-access-5dv9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:32.074435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.074368    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" podUID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:00:32.098859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.098828    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dv9m\" (UniqueName: \"kubernetes.io/projected/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-kube-api-access-5dv9m\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:32.098859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.098852    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:32.098859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.098864    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:32.564616 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.564593    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 14:00:32.605140 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.605054    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553121b1-c367-4a01-868c-e84169451981-kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/553121b1-c367-4a01-868c-e84169451981-kube-api-access-c2bcg\") pod \"553121b1-c367-4a01-868c-e84169451981\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") "
Nov 02 14:00:32.605309 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.605290    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/553121b1-c367-4a01-868c-e84169451981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553121b1-c367-4a01-868c-e84169451981-plugins-config-volume\") pod \"553121b1-c367-4a01-868c-e84169451981\" (UID: \"553121b1-c367-4a01-868c-e84169451981\") "
Nov 02 14:00:32.605778 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.605748    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553121b1-c367-4a01-868c-e84169451981-plugins-config-volume" pod "553121b1-c367-4a01-868c-e84169451981" (UID: "553121b1-c367-4a01-868c-e84169451981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:32.606312 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.606092    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553121b1-c367-4a01-868c-e84169451981-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:32.607348 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.607325    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553121b1-c367-4a01-868c-e84169451981-kube-api-access-c2bcg" pod "553121b1-c367-4a01-868c-e84169451981" (UID: "553121b1-c367-4a01-868c-e84169451981"). InnerVolumeSpecName "kube-api-access-c2bcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:32.709174 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.709137    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/553121b1-c367-4a01-868c-e84169451981-kube-api-access-c2bcg\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:32.823762 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.823732    5572 generic.go:356] "Generic (PLEG): container finished" podID="553121b1-c367-4a01-868c-e84169451981" containerID="ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49" exitCode=0
Nov 02 14:00:32.823909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.823781    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"
Nov 02 14:00:32.823909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.823822    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" event={"ID":"553121b1-c367-4a01-868c-e84169451981","Type":"ContainerDied","Data":"ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49"}
Nov 02 14:00:32.823909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.823878    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5" event={"ID":"553121b1-c367-4a01-868c-e84169451981","Type":"ContainerDied","Data":"b3b321898d2f662de0817cc1b5a09c2ccc301a0cd41c640858ee07a398e3293c"}
Nov 02 14:00:32.823909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.823905    5572 scope.go:122] "RemoveContainer" containerID="ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49"
Nov 02 14:00:32.825656 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.825632    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"
Nov 02 14:00:32.825775 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.825641    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk" event={"ID":"1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa","Type":"ContainerDied","Data":"faee591fdd54e500c4a423cc260ad3a2e259b0f8e9ee292a906e11acf1759e57"}
Nov 02 14:00:32.825775 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.825689    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faee591fdd54e500c4a423cc260ad3a2e259b0f8e9ee292a906e11acf1759e57"
Nov 02 14:00:32.836109 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.836093    5572 scope.go:122] "RemoveContainer" containerID="ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49"
Nov 02 14:00:32.836376 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:32.836353    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49\": container with ID starting with ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49 not found: ID does not exist" containerID="ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49"
Nov 02 14:00:32.836424 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.836385    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49"} err="failed to get container status \"ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49\": rpc error: code = NotFound desc = could not find container \"ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49\": container with ID starting with ad973935779673821b27e3f7275cf2b0f37b323e14e51f1da77b786397f4bb49 not found: ID does not exist"
Nov 02 14:00:32.890027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.889970    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"]
Nov 02 14:00:32.903495 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.903465    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9e0c2f-6c74f676fc-vxpv5"]
Nov 02 14:00:32.988972 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:32.988945    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"]
Nov 02 14:00:33.007628 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:33.007602    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"]
Nov 02 14:00:33.542096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:33.542064    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="553121b1-c367-4a01-868c-e84169451981" path="/var/lib/kubelet/pods/553121b1-c367-4a01-868c-e84169451981/volumes"
Nov 02 14:00:35.297880 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:35.297848    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"]
Nov 02 14:00:35.298376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:35.298094    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy" containerID="cri-o://bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e" gracePeriod=30
Nov 02 14:00:35.332364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:35.332334    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"]
Nov 02 14:00:35.332876 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:35.332842    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy" containerID="cri-o://7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a" gracePeriod=30
Nov 02 14:00:35.346878 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:35.346853    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"]
Nov 02 14:00:35.347047 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:35.347026    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy" containerID="cri-o://6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b" gracePeriod=30
Nov 02 14:00:38.592788 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.592760    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"]
Nov 02 14:00:38.783962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.783940    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 14:00:38.843678 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.843652    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d015852-ebcd-4cc8-9e80-6a025a410d7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d015852-ebcd-4cc8-9e80-6a025a410d7a-plugins-config-volume\") pod \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") "
Nov 02 14:00:38.843802 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.843753    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d015852-ebcd-4cc8-9e80-6a025a410d7a-kube-api-access-9hlw9\" (UniqueName: \"kubernetes.io/projected/3d015852-ebcd-4cc8-9e80-6a025a410d7a-kube-api-access-9hlw9\") pod \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\" (UID: \"3d015852-ebcd-4cc8-9e80-6a025a410d7a\") "
Nov 02 14:00:38.843961 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.843941    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d015852-ebcd-4cc8-9e80-6a025a410d7a-plugins-config-volume" pod "3d015852-ebcd-4cc8-9e80-6a025a410d7a" (UID: "3d015852-ebcd-4cc8-9e80-6a025a410d7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:38.844132 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.844118    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d015852-ebcd-4cc8-9e80-6a025a410d7a-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:38.846006 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.845974    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d015852-ebcd-4cc8-9e80-6a025a410d7a-kube-api-access-9hlw9" pod "3d015852-ebcd-4cc8-9e80-6a025a410d7a" (UID: "3d015852-ebcd-4cc8-9e80-6a025a410d7a"). InnerVolumeSpecName "kube-api-access-9hlw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:38.997964 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:38.990542    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlw9\" (UniqueName: \"kubernetes.io/projected/3d015852-ebcd-4cc8-9e80-6a025a410d7a-kube-api-access-9hlw9\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:39.025210 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.021576    5572 generic.go:356] "Generic (PLEG): container finished" podID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" containerID="a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81" exitCode=0
Nov 02 14:00:39.026621 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.024935    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"
Nov 02 14:00:39.027085 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.024962    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" event={"ID":"3d015852-ebcd-4cc8-9e80-6a025a410d7a","Type":"ContainerDied","Data":"a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81"}
Nov 02 14:00:39.027085 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.026921    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf" event={"ID":"3d015852-ebcd-4cc8-9e80-6a025a410d7a","Type":"ContainerDied","Data":"fec59d6f84b1a5d92f9e086012bb3597c4cf9ffb269411e6dde1f762e0aef344"}
Nov 02 14:00:39.027085 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.026956    5572 scope.go:122] "RemoveContainer" containerID="a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81"
Nov 02 14:00:39.058808 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.058762    5572 scope.go:122] "RemoveContainer" containerID="a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81"
Nov 02 14:00:39.059152 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:39.059128    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81\": container with ID starting with a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81 not found: ID does not exist" containerID="a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81"
Nov 02 14:00:39.059224 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.059161    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81"} err="failed to get container status \"a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81\": rpc error: code = NotFound desc = could not find container \"a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81\": container with ID starting with a64e87f568cd007a40a87d1de47a7a05258eaea578c152a7bffea8e18f92af81 not found: ID does not exist"
Nov 02 14:00:39.068318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.068294    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"]
Nov 02 14:00:39.074057 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.074033    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a9443-5fd9c46cc5-wxggf"]
Nov 02 14:00:39.602490 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:39.602417    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" path="/var/lib/kubelet/pods/3d015852-ebcd-4cc8-9e80-6a025a410d7a/volumes"
Nov 02 14:00:40.624294 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.624271    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 14:00:40.722962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.722928    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istiod-ca-cert\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.728396 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.723688    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istiod-ca-cert" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:40.728735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.728711    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-kube-api-access-fzbb5\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-kube-api-access-fzbb5\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.728901 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.728886    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-credential-socket\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.729120 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.729094    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-token\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-token\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.729247 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.729234    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-podinfo\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.729438 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.729417    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-certs\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.729627 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.729612    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-envoy\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.729767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.729753    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-data\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.729914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.729900    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-socket\") pod \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\" (UID: \"3b87a981-ea31-4e9d-a355-eb7f9ba33924\") "
Nov 02 14:00:40.730901 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.730882    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.731499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.731483    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-data" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.732132 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.732097    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-socket" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.732401 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.732376    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-credential-socket" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.732401 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.732387    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-podinfo" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:40.732578 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.732397    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-certs" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.733138 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.733109    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-envoy" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.734540 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.734516    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-kube-api-access-fzbb5" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "kube-api-access-fzbb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:40.734914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.734889    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 14:00:40.735209 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.735188    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-token" pod "3b87a981-ea31-4e9d-a355-eb7f9ba33924" (UID: "3b87a981-ea31-4e9d-a355-eb7f9ba33924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:40.737601 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.737575    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 14:00:40.777932 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.777909    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"]
Nov 02 14:00:40.778408 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778394    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778410    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778418    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" containerName="payload-processing"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778423    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" containerName="payload-processing"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778433    5572 cpu_manager.go:414] "containerMap: removing container" podUID="1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" containerName="revoke-keys"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778438    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" containerName="revoke-keys"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778449    5572 cpu_manager.go:414] "containerMap: removing container" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778453    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778463    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy"
Nov 02 14:00:40.778464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778468    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778479    5572 cpu_manager.go:414] "containerMap: removing container" podUID="553121b1-c367-4a01-868c-e84169451981" containerName="payload-processing"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778484    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553121b1-c367-4a01-868c-e84169451981" containerName="payload-processing"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778494    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778499    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778506    5572 cpu_manager.go:414] "containerMap: removing container" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778511    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778604    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778611    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d015852-ebcd-4cc8-9e80-6a025a410d7a" containerName="payload-processing"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778617    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc76769-59a3-40d0-9b4e-bf5106d571e7" containerName="cleanup"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778622    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7156d74-04c2-4218-8782-4829ae754bfa" containerName="cleanup"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778628    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" containerName="revoke-keys"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778634    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerName="istio-proxy"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778640    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerName="istio-proxy"
Nov 02 14:00:40.778785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.778645    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="553121b1-c367-4a01-868c-e84169451981" containerName="payload-processing"
Nov 02 14:00:40.782411 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.782390    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.789171 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.788718    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"]
Nov 02 14:00:40.789526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.789506    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerName="istio-proxy"
Nov 02 14:00:40.795748 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.795526    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"]
Nov 02 14:00:40.795748 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.795647    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.796288 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.796220    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:40.796439 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.796416    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:00:40.807837 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.807806    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"]
Nov 02 14:00:40.831586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831544    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-kube-api-access-b8nnl\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-kube-api-access-b8nnl\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.831731 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831632    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-podinfo\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.831731 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831661    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-data\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.831731 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831696    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-token\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-token\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.831731 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831727    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istiod-ca-cert\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.831959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831763    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-certs\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.831959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831795    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-certs\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.831959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831838    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-envoy\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.831959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831881    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-credential-socket\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.831959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831913    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce05724-741f-45ba-bb1e-dce977afe40d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce05724-741f-45ba-bb1e-dce977afe40d-istiod-ca-cert\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.832222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831967    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-kube-api-access-4nncv\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-kube-api-access-4nncv\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.832222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.831996    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-data" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.832222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832002    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-credential-socket\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.832222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832073    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-socket\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.832222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832170    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-token\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-token\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.832222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832186    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-credential-socket" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.832537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832224    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-podinfo\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.832537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832270    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-data\") pod \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\" (UID: \"af70e8d4-a1b3-43de-a8e4-f64df1a61613\") "
Nov 02 14:00:40.832537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832306    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-envoy\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.832537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832361    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-socket\") pod \"2ce05724-741f-45ba-bb1e-dce977afe40d\" (UID: \"2ce05724-741f-45ba-bb1e-dce977afe40d\") "
Nov 02 14:00:40.832537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832422    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce05724-741f-45ba-bb1e-dce977afe40d-istiod-ca-cert" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:40.832537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832499    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.832919 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832607    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.832919 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832651    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-credential-socket" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.832919 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832773    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.832919 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832839    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7sk7\" (UniqueName: \"kubernetes.io/projected/08b957e4-a089-4e30-9750-7fcf7443a881-kube-api-access-d7sk7\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.833136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833049    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.833136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832777    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-certs" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.833136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.832847    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-socket" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.833136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833054    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istiod-ca-cert" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:40.833136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833080    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7pl6\" (UniqueName: \"kubernetes.io/projected/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-kube-api-access-r7pl6\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833220    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833240    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833258    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzbb5\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-kube-api-access-fzbb5\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833275    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833287    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833299    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833312    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833321    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833329    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833341    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833354    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce05724-741f-45ba-bb1e-dce977afe40d-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833365    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833373    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833381    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.833392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833389    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b87a981-ea31-4e9d-a355-eb7f9ba33924-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.834194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833552    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-socket" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.834194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833770    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-certs" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.834194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.833955    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-data" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.836117 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.836084    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-podinfo" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:40.839667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.837765    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-kube-api-access-4nncv" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "kube-api-access-4nncv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:40.839667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.837766    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-envoy" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.839667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.837796    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-podinfo" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:40.839667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.837817    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-token" pod "af70e8d4-a1b3-43de-a8e4-f64df1a61613" (UID: "af70e8d4-a1b3-43de-a8e4-f64df1a61613"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:40.839667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.837869    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-token" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:40.839667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.838243    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-envoy" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:40.839667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.838357    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-kube-api-access-b8nnl" pod "2ce05724-741f-45ba-bb1e-dce977afe40d" (UID: "2ce05724-741f-45ba-bb1e-dce977afe40d"). InnerVolumeSpecName "kube-api-access-b8nnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:40.933948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.933864    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.933948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.933918    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7sk7\" (UniqueName: \"kubernetes.io/projected/08b957e4-a089-4e30-9750-7fcf7443a881-kube-api-access-d7sk7\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.934152 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934072    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.934152 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934101    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7pl6\" (UniqueName: \"kubernetes.io/projected/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-kube-api-access-r7pl6\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.934152 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934126    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.934323 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934182    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.934323 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934291    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8nnl\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-kube-api-access-b8nnl\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934323 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934309    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934330    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934344    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934359    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934373    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nncv\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-kube-api-access-4nncv\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934387    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934402    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934417    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af70e8d4-a1b3-43de-a8e4-f64df1a61613-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934430    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934469 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934445    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce05724-741f-45ba-bb1e-dce977afe40d-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:40.934936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.934650    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.935150 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.935124    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.937484 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.937458    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.937864 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.937839    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.962168 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.962133    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7sk7\" (UniqueName: \"kubernetes.io/projected/08b957e4-a089-4e30-9750-7fcf7443a881-kube-api-access-d7sk7\") pod \"maas-api-revoke-sub-0624fc49-zx8hb\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:40.966353 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.966324    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7pl6\" (UniqueName: \"kubernetes.io/projected/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-kube-api-access-r7pl6\") pod \"maas-api-revoke-sub-aefef8e6-zpd8k\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:40.983582 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.979628    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"]
Nov 02 14:00:40.983582 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.979850    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" containerID="cri-o://71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b" gracePeriod=30
Nov 02 14:00:40.987421 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.986541    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"]
Nov 02 14:00:40.987421 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.986766    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy" containerID="cri-o://b88fe6a51005f0e04a249e3546ed40d68ae33ea5ebd8aee1ad91feeaeefea359" gracePeriod=30
Nov 02 14:00:40.994910 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.994886    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"]
Nov 02 14:00:40.995072 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.995055    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" containerID="cri-o://747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362" gracePeriod=30
Nov 02 14:00:40.999905 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:40.999883    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"]
Nov 02 14:00:41.000061 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.000044    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy" containerID="cri-o://d6e249ce398629d2ea1a58c5399e01eddb3895ebd329f9981edbf1b449e89a3d" gracePeriod=30
Nov 02 14:00:41.095450 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.095416    5572 generic.go:356] "Generic (PLEG): container finished" podID="2ce05724-741f-45ba-bb1e-dce977afe40d" containerID="bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e" exitCode=0
Nov 02 14:00:41.095661 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.095459    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"
Nov 02 14:00:41.095661 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.095507    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" event={"ID":"2ce05724-741f-45ba-bb1e-dce977afe40d","Type":"ContainerDied","Data":"bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e"}
Nov 02 14:00:41.095661 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.095588    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89" event={"ID":"2ce05724-741f-45ba-bb1e-dce977afe40d","Type":"ContainerDied","Data":"471821754c0cad07c30e70fec6e8dabed751b5d6e7bca136c43fc4bee7754b51"}
Nov 02 14:00:41.095661 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.095616    5572 scope.go:122] "RemoveContainer" containerID="bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e"
Nov 02 14:00:41.125055 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.118756    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:41.125055 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.119192    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:41.131222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.130914    5572 generic.go:356] "Generic (PLEG): container finished" podID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" containerID="6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b" exitCode=0
Nov 02 14:00:41.131222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.131107    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" event={"ID":"af70e8d4-a1b3-43de-a8e4-f64df1a61613","Type":"ContainerDied","Data":"6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b"}
Nov 02 14:00:41.131222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.131161    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm" event={"ID":"af70e8d4-a1b3-43de-a8e4-f64df1a61613","Type":"ContainerDied","Data":"47884108c8799d8808593e5b288a49dc911921c426be6dc91532ca3b0b97bf2c"}
Nov 02 14:00:41.131531 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.131305    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"
Nov 02 14:00:41.182129 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.174974    5572 generic.go:356] "Generic (PLEG): container finished" podID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" containerID="7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a" exitCode=0
Nov 02 14:00:41.182129 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.175031    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" event={"ID":"3b87a981-ea31-4e9d-a355-eb7f9ba33924","Type":"ContainerDied","Data":"7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a"}
Nov 02 14:00:41.182129 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.175078    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt" event={"ID":"3b87a981-ea31-4e9d-a355-eb7f9ba33924","Type":"ContainerDied","Data":"4228d0d1c29622e9f73720f2c253a501f1547e01b7db45bb70bdcd9e94f1b6b7"}
Nov 02 14:00:41.182129 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.175175    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"
Nov 02 14:00:41.183587 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.183542    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"]
Nov 02 14:00:41.196328 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.195470    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-gqq89"]
Nov 02 14:00:41.205702 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.205684    5572 scope.go:122] "RemoveContainer" containerID="f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40"
Nov 02 14:00:41.218618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.218601    5572 scope.go:122] "RemoveContainer" containerID="bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e"
Nov 02 14:00:41.218967 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:41.218944    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e\": container with ID starting with bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e not found: ID does not exist" containerID="bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e"
Nov 02 14:00:41.219040 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.218978    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e"} err="failed to get container status \"bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e\": rpc error: code = NotFound desc = could not find container \"bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e\": container with ID starting with bbaa4c0914365326ccdd3de8b3296aff0c43902f14ca4fe523242fdbf4cfc87e not found: ID does not exist"
Nov 02 14:00:41.219040 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.218997    5572 scope.go:122] "RemoveContainer" containerID="f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40"
Nov 02 14:00:41.219283 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:41.219255    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40\": container with ID starting with f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40 not found: ID does not exist" containerID="f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40"
Nov 02 14:00:41.219384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.219284    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40"} err="failed to get container status \"f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40\": rpc error: code = NotFound desc = could not find container \"f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40\": container with ID starting with f7676408084fe0bdbe32d3bd2dc72ea55d51c9fc6d3c7ab8d819ffa24c6c2e40 not found: ID does not exist"
Nov 02 14:00:41.219384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.219301    5572 scope.go:122] "RemoveContainer" containerID="6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b"
Nov 02 14:00:41.232302 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.232271    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"]
Nov 02 14:00:41.237302 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.237266    5572 scope.go:122] "RemoveContainer" containerID="6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b"
Nov 02 14:00:41.237776 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:41.237725    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b\": container with ID starting with 6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b not found: ID does not exist" containerID="6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b"
Nov 02 14:00:41.237909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.237785    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b"} err="failed to get container status \"6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b\": rpc error: code = NotFound desc = could not find container \"6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b\": container with ID starting with 6a96d7c7fa6054b2a5d423af95476af73e1371ef80df08728fa026c7adafc97b not found: ID does not exist"
Nov 02 14:00:41.237909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.237811    5572 scope.go:122] "RemoveContainer" containerID="7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a"
Nov 02 14:00:41.241896 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.241874    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-d8wdm"]
Nov 02 14:00:41.257823 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.257802    5572 scope.go:122] "RemoveContainer" containerID="7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a"
Nov 02 14:00:41.258251 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:41.258223    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a\": container with ID starting with 7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a not found: ID does not exist" containerID="7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a"
Nov 02 14:00:41.258336 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.258262    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a"} err="failed to get container status \"7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a\": rpc error: code = NotFound desc = could not find container \"7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a\": container with ID starting with 7ae1d6b3a660be64135ea1003c719573b2e84eab863df7f6d0201febdef8277a not found: ID does not exist"
Nov 02 14:00:41.261129 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.261101    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"]
Nov 02 14:00:41.274459 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.274132    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9e0c2f-openshift-default-55ccb4cb79-wlzjt"]
Nov 02 14:00:41.499532 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:00:41.499497    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc09e29_f0f5_47c7_9419_0278ceef1f7e.slice/crio-93db026e81fefc06b9c21b4fab41b7e70e0964832205298ad4d7eb725ef04c8f WatchSource:0}: Error finding container 93db026e81fefc06b9c21b4fab41b7e70e0964832205298ad4d7eb725ef04c8f: Status 404 returned error can't find the container with id 93db026e81fefc06b9c21b4fab41b7e70e0964832205298ad4d7eb725ef04c8f
Nov 02 14:00:41.499700 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.499595    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"]
Nov 02 14:00:41.510683 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.510660    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"]
Nov 02 14:00:41.514856 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:00:41.514833    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b957e4_a089_4e30_9750_7fcf7443a881.slice/crio-80e5705e9ac25412c728cc85a25a67c24d71cdf4e8020824d4eb4860735ffadd WatchSource:0}: Error finding container 80e5705e9ac25412c728cc85a25a67c24d71cdf4e8020824d4eb4860735ffadd: Status 404 returned error can't find the container with id 80e5705e9ac25412c728cc85a25a67c24d71cdf4e8020824d4eb4860735ffadd
Nov 02 14:00:41.542781 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.542752    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce05724-741f-45ba-bb1e-dce977afe40d" path="/var/lib/kubelet/pods/2ce05724-741f-45ba-bb1e-dce977afe40d/volumes"
Nov 02 14:00:41.543307 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.543291    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b87a981-ea31-4e9d-a355-eb7f9ba33924" path="/var/lib/kubelet/pods/3b87a981-ea31-4e9d-a355-eb7f9ba33924/volumes"
Nov 02 14:00:41.543731 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:41.543716    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af70e8d4-a1b3-43de-a8e4-f64df1a61613" path="/var/lib/kubelet/pods/af70e8d4-a1b3-43de-a8e4-f64df1a61613/volumes"
Nov 02 14:00:42.186648 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:42.186610    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb" event={"ID":"08b957e4-a089-4e30-9750-7fcf7443a881","Type":"ContainerStarted","Data":"f3c5ab0b14fca07937fd965647e990d85cd8fb41d657a0174120953862c59802"}
Nov 02 14:00:42.187096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:42.186694    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb" event={"ID":"08b957e4-a089-4e30-9750-7fcf7443a881","Type":"ContainerStarted","Data":"80e5705e9ac25412c728cc85a25a67c24d71cdf4e8020824d4eb4860735ffadd"}
Nov 02 14:00:42.193204 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:42.193170    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k" event={"ID":"bcc09e29-f0f5-47c7-9419-0278ceef1f7e","Type":"ContainerStarted","Data":"e701253ff101ddfb9040de432f1b675b44264a21475849b6b635acbbeee369e4"}
Nov 02 14:00:42.193371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:42.193225    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k" event={"ID":"bcc09e29-f0f5-47c7-9419-0278ceef1f7e","Type":"ContainerStarted","Data":"93db026e81fefc06b9c21b4fab41b7e70e0964832205298ad4d7eb725ef04c8f"}
Nov 02 14:00:42.209803 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:42.209760    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb" podStartSLOduration=2.209747281 podStartE2EDuration="2.209747281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:42.207067041 +0000 UTC m=+1018.785155909" watchObservedRunningTime="2026-11-02 14:00:42.209747281 +0000 UTC m=+1018.787836149"
Nov 02 14:00:42.232574 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:42.232519    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k" podStartSLOduration=2.232506511 podStartE2EDuration="2.232506511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:42.228114389 +0000 UTC m=+1018.806203257" watchObservedRunningTime="2026-11-02 14:00:42.232506511 +0000 UTC m=+1018.810595377"
Nov 02 14:00:43.205603 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:43.205547    5572 generic.go:356] "Generic (PLEG): container finished" podID="08b957e4-a089-4e30-9750-7fcf7443a881" containerID="f3c5ab0b14fca07937fd965647e990d85cd8fb41d657a0174120953862c59802" exitCode=0
Nov 02 14:00:43.205603 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:43.205595    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb" event={"ID":"08b957e4-a089-4e30-9750-7fcf7443a881","Type":"ContainerDied","Data":"f3c5ab0b14fca07937fd965647e990d85cd8fb41d657a0174120953862c59802"}
Nov 02 14:00:43.207349 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:43.207327    5572 generic.go:356] "Generic (PLEG): container finished" podID="bcc09e29-f0f5-47c7-9419-0278ceef1f7e" containerID="e701253ff101ddfb9040de432f1b675b44264a21475849b6b635acbbeee369e4" exitCode=0
Nov 02 14:00:43.207478 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:43.207402    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k" event={"ID":"bcc09e29-f0f5-47c7-9419-0278ceef1f7e","Type":"ContainerDied","Data":"e701253ff101ddfb9040de432f1b675b44264a21475849b6b635acbbeee369e4"}
Nov 02 14:00:43.593783 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:43.593748    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" podUID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" containerName="payload-processing" containerID="cri-o://46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613" gracePeriod=25
Nov 02 14:00:43.712376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:43.712347    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:00:43.712509 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:43.712397    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:00:44.443688 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.441123    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:44.510635 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.496320    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-kube-api-access-r7pl6\" (UniqueName: \"kubernetes.io/projected/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-kube-api-access-r7pl6\") pod \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") "
Nov 02 14:00:44.510635 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.496443    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-openshift-ca\") pod \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") "
Nov 02 14:00:44.510635 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.496479    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-xks-ca\") pod \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\" (UID: \"bcc09e29-f0f5-47c7-9419-0278ceef1f7e\") "
Nov 02 14:00:44.533873 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.524116    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-kube-api-access-r7pl6" pod "bcc09e29-f0f5-47c7-9419-0278ceef1f7e" (UID: "bcc09e29-f0f5-47c7-9419-0278ceef1f7e"). InnerVolumeSpecName "kube-api-access-r7pl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:44.533873 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.524447    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-openshift-ca" pod "bcc09e29-f0f5-47c7-9419-0278ceef1f7e" (UID: "bcc09e29-f0f5-47c7-9419-0278ceef1f7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:44.534361 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.534328    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-xks-ca" pod "bcc09e29-f0f5-47c7-9419-0278ceef1f7e" (UID: "bcc09e29-f0f5-47c7-9419-0278ceef1f7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:44.549791 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.549760    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:44.598111 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.598080    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7pl6\" (UniqueName: \"kubernetes.io/projected/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-kube-api-access-r7pl6\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:44.598111 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.598113    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:44.598307 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.598126    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc09e29-f0f5-47c7-9419-0278ceef1f7e-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:44.699147 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.699118    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b957e4-a089-4e30-9750-7fcf7443a881-kube-api-access-d7sk7\" (UniqueName: \"kubernetes.io/projected/08b957e4-a089-4e30-9750-7fcf7443a881-kube-api-access-d7sk7\") pod \"08b957e4-a089-4e30-9750-7fcf7443a881\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") "
Nov 02 14:00:44.699354 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.699163    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-openshift-ca\") pod \"08b957e4-a089-4e30-9750-7fcf7443a881\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") "
Nov 02 14:00:44.699354 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.699268    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-xks-ca\") pod \"08b957e4-a089-4e30-9750-7fcf7443a881\" (UID: \"08b957e4-a089-4e30-9750-7fcf7443a881\") "
Nov 02 14:00:44.699539 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.699517    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-openshift-ca" pod "08b957e4-a089-4e30-9750-7fcf7443a881" (UID: "08b957e4-a089-4e30-9750-7fcf7443a881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:44.699727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.699711    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:44.701397 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.701377    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-xks-ca" pod "08b957e4-a089-4e30-9750-7fcf7443a881" (UID: "08b957e4-a089-4e30-9750-7fcf7443a881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:44.701445 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.701406    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b957e4-a089-4e30-9750-7fcf7443a881-kube-api-access-d7sk7" pod "08b957e4-a089-4e30-9750-7fcf7443a881" (UID: "08b957e4-a089-4e30-9750-7fcf7443a881"). InnerVolumeSpecName "kube-api-access-d7sk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:44.800409 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.800328    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b957e4-a089-4e30-9750-7fcf7443a881-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:44.800409 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:44.800356    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7sk7\" (UniqueName: \"kubernetes.io/projected/08b957e4-a089-4e30-9750-7fcf7443a881-kube-api-access-d7sk7\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:45.274611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.274543    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"
Nov 02 14:00:45.274789 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.274741    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb" event={"ID":"08b957e4-a089-4e30-9750-7fcf7443a881","Type":"ContainerDied","Data":"80e5705e9ac25412c728cc85a25a67c24d71cdf4e8020824d4eb4860735ffadd"}
Nov 02 14:00:45.274889 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.274850    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e5705e9ac25412c728cc85a25a67c24d71cdf4e8020824d4eb4860735ffadd"
Nov 02 14:00:45.278502 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.278473    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k" event={"ID":"bcc09e29-f0f5-47c7-9419-0278ceef1f7e","Type":"ContainerDied","Data":"93db026e81fefc06b9c21b4fab41b7e70e0964832205298ad4d7eb725ef04c8f"}
Nov 02 14:00:45.278646 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.278524    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93db026e81fefc06b9c21b4fab41b7e70e0964832205298ad4d7eb725ef04c8f"
Nov 02 14:00:45.278646 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.278614    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"
Nov 02 14:00:45.340045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.340013    5572 patch_prober.go:29] interesting pod/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:00:45.340232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:45.340072    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:00:46.260003 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.259652    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 14:00:46.280430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.280403    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 14:00:46.291990 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.291788    5572 generic.go:356] "Generic (PLEG): container finished" podID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerID="b88fe6a51005f0e04a249e3546ed40d68ae33ea5ebd8aee1ad91feeaeefea359" exitCode=0
Nov 02 14:00:46.291990 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.291899    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" event={"ID":"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c","Type":"ContainerDied","Data":"b88fe6a51005f0e04a249e3546ed40d68ae33ea5ebd8aee1ad91feeaeefea359"}
Nov 02 14:00:46.294642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.294444    5572 generic.go:356] "Generic (PLEG): container finished" podID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerID="d6e249ce398629d2ea1a58c5399e01eddb3895ebd329f9981edbf1b449e89a3d" exitCode=0
Nov 02 14:00:46.294642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.294543    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" event={"ID":"421a53e0-3b6f-42a5-8256-71e356f87c49","Type":"ContainerDied","Data":"d6e249ce398629d2ea1a58c5399e01eddb3895ebd329f9981edbf1b449e89a3d"}
Nov 02 14:00:46.296702 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.296273    5572 generic.go:356] "Generic (PLEG): container finished" podID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerID="71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b" exitCode=0
Nov 02 14:00:46.296702 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.296386    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" event={"ID":"651ba3ec-24ee-4e08-8c41-e77066d4dd96","Type":"ContainerDied","Data":"71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b"}
Nov 02 14:00:46.296702 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.296422    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr" event={"ID":"651ba3ec-24ee-4e08-8c41-e77066d4dd96","Type":"ContainerDied","Data":"cdbc9ffeac4d012bba6661e91f2cee46f1f31cf46a7926723f520de6e00c9518"}
Nov 02 14:00:46.296702 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.296453    5572 scope.go:122] "RemoveContainer" containerID="71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b"
Nov 02 14:00:46.296702 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.296652    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"
Nov 02 14:00:46.299805 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.299780    5572 generic.go:356] "Generic (PLEG): container finished" podID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerID="747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362" exitCode=0
Nov 02 14:00:46.299909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.299845    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" event={"ID":"2f7c381f-262b-47f1-a570-1dd247e0955f","Type":"ContainerDied","Data":"747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362"}
Nov 02 14:00:46.299978 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.299926    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg" event={"ID":"2f7c381f-262b-47f1-a570-1dd247e0955f","Type":"ContainerDied","Data":"e2cd525d4637a984d35afc14a3e1ecad8707006d7a89147d37798d48fd458cd4"}
Nov 02 14:00:46.300099 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.300076    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"
Nov 02 14:00:46.319608 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.319479    5572 scope.go:122] "RemoveContainer" containerID="71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b"
Nov 02 14:00:46.320290 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:46.320235    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b\": container with ID starting with 71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b not found: ID does not exist" containerID="71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b"
Nov 02 14:00:46.320290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.320272    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b"} err="failed to get container status \"71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b\": rpc error: code = NotFound desc = could not find container \"71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b\": container with ID starting with 71dc019dde1a42599ceb3ae46bba31c2ac93a527d1d0d80e6e8638461a05820b not found: ID does not exist"
Nov 02 14:00:46.320290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.320293    5572 scope.go:122] "RemoveContainer" containerID="747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362"
Nov 02 14:00:46.332783 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.332758    5572 scope.go:122] "RemoveContainer" containerID="747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362"
Nov 02 14:00:46.333283 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:46.333255    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362\": container with ID starting with 747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362 not found: ID does not exist" containerID="747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362"
Nov 02 14:00:46.333371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.333295    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362"} err="failed to get container status \"747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362\": rpc error: code = NotFound desc = could not find container \"747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362\": container with ID starting with 747de2f6d1cc99822ef1c8cd7f9ed2b8777c42eadaf0c7b5b730619a2c322362 not found: ID does not exist"
Nov 02 14:00:46.342823 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.342797    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 14:00:46.353463 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.353446    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 14:00:46.360769 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.360738    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-data\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.360886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.360790    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istiod-ca-cert\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.360886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.360809    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-podinfo\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.360886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.360827    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-envoy\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.360886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.360878    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-certs\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.360997    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-credential-socket\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361009    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-data" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.361096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361061    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-certs\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.361096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361092    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-kube-api-access-p2zrz\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-kube-api-access-p2zrz\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361131    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-podinfo\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361141    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-certs" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.361299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361166    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-token\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-token\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361219    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-socket\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361259    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-credential-socket\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.361299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361287    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-data\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361217    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istiod-ca-cert" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:46.361641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361307    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-credential-socket" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.361641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361506    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-socket" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.361641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361325    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-envoy\") pod \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\" (UID: \"651ba3ec-24ee-4e08-8c41-e77066d4dd96\") "
Nov 02 14:00:46.361641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361601    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-socket\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.361898 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361647    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7c381f-262b-47f1-a570-1dd247e0955f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7c381f-262b-47f1-a570-1dd247e0955f-istiod-ca-cert\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.361898 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361698    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-token\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-token\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361505    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-certs" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.361815    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-credential-socket" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362038    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-kube-api-access-nxb72\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-kube-api-access-nxb72\") pod \"2f7c381f-262b-47f1-a570-1dd247e0955f\" (UID: \"2f7c381f-262b-47f1-a570-1dd247e0955f\") "
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362188    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-data" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362623    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362648    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362663    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362679    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362692    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362707    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362721    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.362735    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.363327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.363042    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7c381f-262b-47f1-a570-1dd247e0955f-istiod-ca-cert" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:46.364097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.363682    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-envoy" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.364097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.363726    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-socket" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.364097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.363747    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-token" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.364433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.364233    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-kube-api-access-p2zrz" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "kube-api-access-p2zrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.364433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.364303    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-envoy" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.364610 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.364589    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-kube-api-access-nxb72" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "kube-api-access-nxb72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.364772 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.364755    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-podinfo" pod "651ba3ec-24ee-4e08-8c41-e77066d4dd96" (UID: "651ba3ec-24ee-4e08-8c41-e77066d4dd96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:46.365609 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.365591    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-token" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.365687 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.365667    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-podinfo" pod "2f7c381f-262b-47f1-a570-1dd247e0955f" (UID: "2f7c381f-262b-47f1-a570-1dd247e0955f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:46.463726 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463690    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-podinfo\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.463887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463746    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-data\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.463887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463767    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-credential-socket\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.463887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463790    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-socket\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.463887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463808    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-certs\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.463887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463823    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-envoy\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.463887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463844    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-kube-api-access-rlbs4\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-kube-api-access-rlbs4\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.463887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463883    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421a53e0-3b6f-42a5-8256-71e356f87c49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421a53e0-3b6f-42a5-8256-71e356f87c49-istiod-ca-cert\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463912    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-podinfo\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463938    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-token\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-token\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.463980    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-envoy\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464027    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-credential-socket\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464077    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-certs\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464134    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-socket\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464166    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-token\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464191    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-kube-api-access-2mb87\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-kube-api-access-2mb87\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464236    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istiod-ca-cert\") pod \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\" (UID: \"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c\") "
Nov 02 14:00:46.464868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464264    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-data\") pod \"421a53e0-3b6f-42a5-8256-71e356f87c49\" (UID: \"421a53e0-3b6f-42a5-8256-71e356f87c49\") "
Nov 02 14:00:46.464868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464794    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2zrz\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-kube-api-access-p2zrz\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.464868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464817    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.464868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464835    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.464868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464849    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651ba3ec-24ee-4e08-8c41-e77066d4dd96-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.464868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464023    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-data" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.464868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464610    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421a53e0-3b6f-42a5-8256-71e356f87c49-istiod-ca-cert" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464863    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464044    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-credential-socket" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464265    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-certs" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464278    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-socket" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464482    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-socket" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464854    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-credential-socket" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464906    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7c381f-262b-47f1-a570-1dd247e0955f-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464946    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464967    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxb72\" (UniqueName: \"kubernetes.io/projected/2f7c381f-262b-47f1-a570-1dd247e0955f-kube-api-access-nxb72\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464985    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.464999    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7c381f-262b-47f1-a570-1dd247e0955f-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.465050    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-data" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.465225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.465112    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-certs" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.465879 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.465536    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istiod-ca-cert" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:46.467590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.466918    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-podinfo" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:46.467590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.467353    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-podinfo" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:00:46.467590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.467480    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-token" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.468090 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.468054    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-envoy" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.468090 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.468178    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-envoy" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:46.468090 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.468256    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-kube-api-access-2mb87" pod "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" (UID: "7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c"). InnerVolumeSpecName "kube-api-access-2mb87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.468468 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.468446    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-kube-api-access-rlbs4" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "kube-api-access-rlbs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.469121 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.469099    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-token" pod "421a53e0-3b6f-42a5-8256-71e356f87c49" (UID: "421a53e0-3b6f-42a5-8256-71e356f87c49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:46.566256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566225    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566257    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mb87\" (UniqueName: \"kubernetes.io/projected/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-kube-api-access-2mb87\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566273    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566287    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566301    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566316    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566331    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566347    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566358    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566367    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566378    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlbs4\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-kube-api-access-rlbs4\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566386    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421a53e0-3b6f-42a5-8256-71e356f87c49-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566394    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566402    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421a53e0-3b6f-42a5-8256-71e356f87c49-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566411    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566419    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566426    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.566705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.566436    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:46.624155 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.624105    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"]
Nov 02 14:00:46.628830 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.628809    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-ttqlr"]
Nov 02 14:00:46.652517 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.652486    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"]
Nov 02 14:00:46.656456 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:46.656431    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-p7hrg"]
Nov 02 14:00:47.308610 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.308581    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"
Nov 02 14:00:47.308610 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.308593    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx" event={"ID":"7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c","Type":"ContainerDied","Data":"614a1bfe1dc7b9935dd4a601124976ea74576980e6436dd1429d06df56c5a2c7"}
Nov 02 14:00:47.309139 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.308667    5572 scope.go:122] "RemoveContainer" containerID="b88fe6a51005f0e04a249e3546ed40d68ae33ea5ebd8aee1ad91feeaeefea359"
Nov 02 14:00:47.310342 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.310272    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"
Nov 02 14:00:47.310342 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.310303    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z" event={"ID":"421a53e0-3b6f-42a5-8256-71e356f87c49","Type":"ContainerDied","Data":"3583706ba8dbcaa5aaf21c71874d0054fb9ff482e5aef0c1e80fa283e524a67b"}
Nov 02 14:00:47.320426 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.320397    5572 scope.go:122] "RemoveContainer" containerID="d6e249ce398629d2ea1a58c5399e01eddb3895ebd329f9981edbf1b449e89a3d"
Nov 02 14:00:47.380893 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.380853    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"]
Nov 02 14:00:47.393970 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.393943    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-qn5wx"]
Nov 02 14:00:47.426318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.426291    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"]
Nov 02 14:00:47.442001 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.441968    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a9443-openshift-default-67958d446f-7zq6z"]
Nov 02 14:00:47.541976 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.541945    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" path="/var/lib/kubelet/pods/2f7c381f-262b-47f1-a570-1dd247e0955f/volumes"
Nov 02 14:00:47.542397 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.542384    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" path="/var/lib/kubelet/pods/421a53e0-3b6f-42a5-8256-71e356f87c49/volumes"
Nov 02 14:00:47.542814 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.542800    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" path="/var/lib/kubelet/pods/651ba3ec-24ee-4e08-8c41-e77066d4dd96/volumes"
Nov 02 14:00:47.543499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:47.543484    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" path="/var/lib/kubelet/pods/7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c/volumes"
Nov 02 14:00:48.009735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.009702    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main" containerID="cri-o://0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b" gracePeriod=45
Nov 02 14:00:48.032031 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.031966    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main" containerID="cri-o://4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf" gracePeriod=45
Nov 02 14:00:48.297447 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.297423    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 14:00:48.322935 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.322903    5572 generic.go:356] "Generic (PLEG): container finished" podID="3120bf68-fa50-4b09-9725-be8ed6134529" containerID="0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b" exitCode=0
Nov 02 14:00:48.323416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.322949    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"
Nov 02 14:00:48.323416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.322996    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" event={"ID":"3120bf68-fa50-4b09-9725-be8ed6134529","Type":"ContainerDied","Data":"0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b"}
Nov 02 14:00:48.323416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.323046    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf" event={"ID":"3120bf68-fa50-4b09-9725-be8ed6134529","Type":"ContainerDied","Data":"d6cf91b32fd79479c7f3b3220cf0b94eb3be64a581ea110eb2275f54124d899a"}
Nov 02 14:00:48.323416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.323062    5572 scope.go:122] "RemoveContainer" containerID="0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b"
Nov 02 14:00:48.323865 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.323830    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 14:00:48.325604 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.324988    5572 generic.go:356] "Generic (PLEG): container finished" podID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerID="4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf" exitCode=0
Nov 02 14:00:48.325604 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.325029    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" event={"ID":"59e91d94-7c4f-4d9e-ac04-04745b267604","Type":"ContainerDied","Data":"4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf"}
Nov 02 14:00:48.325604 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.325065    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m" event={"ID":"59e91d94-7c4f-4d9e-ac04-04745b267604","Type":"ContainerDied","Data":"058e7c9af90ce1310f6268a5f053e545a07369d13ddd9fe508130c74ba601814"}
Nov 02 14:00:48.337719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.337629    5572 scope.go:122] "RemoveContainer" containerID="0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b"
Nov 02 14:00:48.338131 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:48.338105    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b\": container with ID starting with 0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b not found: ID does not exist" containerID="0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b"
Nov 02 14:00:48.338225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.338143    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b"} err="failed to get container status \"0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b\": rpc error: code = NotFound desc = could not find container \"0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b\": container with ID starting with 0ab736a12e07aa9644a03deb2e91b355d02bf7f4279003fb1c325c43496d921b not found: ID does not exist"
Nov 02 14:00:48.338225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.338167    5572 scope.go:122] "RemoveContainer" containerID="4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf"
Nov 02 14:00:48.347603 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.347582    5572 scope.go:122] "RemoveContainer" containerID="4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf"
Nov 02 14:00:48.347971 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:48.347951    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf\": container with ID starting with 4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf not found: ID does not exist" containerID="4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf"
Nov 02 14:00:48.348040 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.347981    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf"} err="failed to get container status \"4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf\": rpc error: code = NotFound desc = could not find container \"4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf\": container with ID starting with 4e0ed610e03578c1c682ab2cb2cc19dc7ba991c375d4ac7627e8a14dd47021cf not found: ID does not exist"
Nov 02 14:00:48.382611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382583    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-model-cache\") pod \"3120bf68-fa50-4b09-9725-be8ed6134529\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") "
Nov 02 14:00:48.382799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382624    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-home\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-home\") pod \"59e91d94-7c4f-4d9e-ac04-04745b267604\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") "
Nov 02 14:00:48.382799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382653    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-tmp-dir\") pod \"3120bf68-fa50-4b09-9725-be8ed6134529\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") "
Nov 02 14:00:48.382799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382690    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-model-cache\") pod \"59e91d94-7c4f-4d9e-ac04-04745b267604\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") "
Nov 02 14:00:48.382799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382730    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-tmp-dir\") pod \"59e91d94-7c4f-4d9e-ac04-04745b267604\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") "
Nov 02 14:00:48.382799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382764    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e91d94-7c4f-4d9e-ac04-04745b267604-kube-api-access-v9mcm\" (UniqueName: \"kubernetes.io/projected/59e91d94-7c4f-4d9e-ac04-04745b267604-kube-api-access-v9mcm\") pod \"59e91d94-7c4f-4d9e-ac04-04745b267604\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") "
Nov 02 14:00:48.382799 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382796    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-home\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-home\") pod \"3120bf68-fa50-4b09-9725-be8ed6134529\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") "
Nov 02 14:00:48.383269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382826    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3120bf68-fa50-4b09-9725-be8ed6134529-tls-certs\" (UniqueName: \"kubernetes.io/secret/3120bf68-fa50-4b09-9725-be8ed6134529-tls-certs\") pod \"3120bf68-fa50-4b09-9725-be8ed6134529\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") "
Nov 02 14:00:48.383269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382906    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-dshm\") pod \"3120bf68-fa50-4b09-9725-be8ed6134529\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") "
Nov 02 14:00:48.383269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.382935    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59e91d94-7c4f-4d9e-ac04-04745b267604-tls-certs\" (UniqueName: \"kubernetes.io/secret/59e91d94-7c4f-4d9e-ac04-04745b267604-tls-certs\") pod \"59e91d94-7c4f-4d9e-ac04-04745b267604\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") "
Nov 02 14:00:48.383269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.383047    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-dshm\") pod \"59e91d94-7c4f-4d9e-ac04-04745b267604\" (UID: \"59e91d94-7c4f-4d9e-ac04-04745b267604\") "
Nov 02 14:00:48.383269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.383093    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3120bf68-fa50-4b09-9725-be8ed6134529-kube-api-access-w7ckx\" (UniqueName: \"kubernetes.io/projected/3120bf68-fa50-4b09-9725-be8ed6134529-kube-api-access-w7ckx\") pod \"3120bf68-fa50-4b09-9725-be8ed6134529\" (UID: \"3120bf68-fa50-4b09-9725-be8ed6134529\") "
Nov 02 14:00:48.384168 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.383994    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-tmp-dir" pod "3120bf68-fa50-4b09-9725-be8ed6134529" (UID: "3120bf68-fa50-4b09-9725-be8ed6134529"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.384283 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.384191    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-model-cache" pod "59e91d94-7c4f-4d9e-ac04-04745b267604" (UID: "59e91d94-7c4f-4d9e-ac04-04745b267604"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.386991 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.386962    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3120bf68-fa50-4b09-9725-be8ed6134529-tls-certs" pod "3120bf68-fa50-4b09-9725-be8ed6134529" (UID: "3120bf68-fa50-4b09-9725-be8ed6134529"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:48.387175 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.387149    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e91d94-7c4f-4d9e-ac04-04745b267604-kube-api-access-v9mcm" pod "59e91d94-7c4f-4d9e-ac04-04745b267604" (UID: "59e91d94-7c4f-4d9e-ac04-04745b267604"). InnerVolumeSpecName "kube-api-access-v9mcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:48.387241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.387157    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-home" pod "3120bf68-fa50-4b09-9725-be8ed6134529" (UID: "3120bf68-fa50-4b09-9725-be8ed6134529"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.387595 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.387546    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3120bf68-fa50-4b09-9725-be8ed6134529-kube-api-access-w7ckx" pod "3120bf68-fa50-4b09-9725-be8ed6134529" (UID: "3120bf68-fa50-4b09-9725-be8ed6134529"). InnerVolumeSpecName "kube-api-access-w7ckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:48.388182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.388157    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-tmp-dir" pod "59e91d94-7c4f-4d9e-ac04-04745b267604" (UID: "59e91d94-7c4f-4d9e-ac04-04745b267604"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.388404 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.388370    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-home" pod "59e91d94-7c4f-4d9e-ac04-04745b267604" (UID: "59e91d94-7c4f-4d9e-ac04-04745b267604"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.388748 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.388730    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-model-cache" pod "3120bf68-fa50-4b09-9725-be8ed6134529" (UID: "3120bf68-fa50-4b09-9725-be8ed6134529"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.389914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.389889    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-dshm" pod "59e91d94-7c4f-4d9e-ac04-04745b267604" (UID: "59e91d94-7c4f-4d9e-ac04-04745b267604"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.391783 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.391757    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e91d94-7c4f-4d9e-ac04-04745b267604-tls-certs" pod "59e91d94-7c4f-4d9e-ac04-04745b267604" (UID: "59e91d94-7c4f-4d9e-ac04-04745b267604"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:48.392886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.392861    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-dshm" pod "3120bf68-fa50-4b09-9725-be8ed6134529" (UID: "3120bf68-fa50-4b09-9725-be8ed6134529"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:00:48.484484 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484449    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484484 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484483    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ckx\" (UniqueName: \"kubernetes.io/projected/3120bf68-fa50-4b09-9725-be8ed6134529-kube-api-access-w7ckx\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484494    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484503    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484512    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484519    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484527    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e91d94-7c4f-4d9e-ac04-04745b267604-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484535    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9mcm\" (UniqueName: \"kubernetes.io/projected/59e91d94-7c4f-4d9e-ac04-04745b267604-kube-api-access-v9mcm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484543    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484552    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3120bf68-fa50-4b09-9725-be8ed6134529-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484588    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3120bf68-fa50-4b09-9725-be8ed6134529-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.484707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.484596    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e91d94-7c4f-4d9e-ac04-04745b267604-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:00:48.648097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.648057    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"]
Nov 02 14:00:48.654286 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:48.654264    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-2-w2-ee7df2-kserve-cdb8c4bfc-p7mdf"]
Nov 02 14:00:49.337372 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:49.337344    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"
Nov 02 14:00:49.361920 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:49.361895    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"]
Nov 02 14:00:49.366481 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:49.366460    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ee7df2/distinct-w2-ee7df2-kserve-79cd6bfdc8-fwc7m"]
Nov 02 14:00:49.542995 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:49.542961    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" path="/var/lib/kubelet/pods/3120bf68-fa50-4b09-9725-be8ed6134529/volumes"
Nov 02 14:00:49.543357 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:49.543344    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" path="/var/lib/kubelet/pods/59e91d94-7c4f-4d9e-ac04-04745b267604/volumes"
Nov 02 14:00:51.069764 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:51.069729    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" podUID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:00:51.429726 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:00:51.429650    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:00:56.034915 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:56.034883    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"]
Nov 02 14:00:56.039091 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:56.039063    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"]
Nov 02 14:00:56.042145 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:56.042131    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299ee5b8-dz7p4"]
Nov 02 14:00:56.045182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:56.045161    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2720b9-2m5rf"]
Nov 02 14:00:56.075367 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:56.075344    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"]
Nov 02 14:00:56.078359 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:56.078341    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ee7df2-plqsk"]
Nov 02 14:00:57.541721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:57.541689    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa" path="/var/lib/kubelet/pods/1cb0eea7-fc3a-464a-96b7-06cbbe12a9fa/volumes"
Nov 02 14:00:57.542177 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:57.542085    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227544da-1f67-49c9-9bd7-289b9328a0ba" path="/var/lib/kubelet/pods/227544da-1f67-49c9-9bd7-289b9328a0ba/volumes"
Nov 02 14:00:57.542435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:57.542421    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b66ac5a-c928-4404-9435-492ad0ca8ea6" path="/var/lib/kubelet/pods/9b66ac5a-c928-4404-9435-492ad0ca8ea6/volumes"
Nov 02 14:00:59.032429 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:59.032391    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"]
Nov 02 14:00:59.036032 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:59.036007    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37850f9c-xxdb6"]
Nov 02 14:00:59.542052 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:00:59.542019    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051d5861-951e-49d9-bda4-0fe5929ced94" path="/var/lib/kubelet/pods/051d5861-951e-49d9-bda4-0fe5929ced94/volumes"
Nov 02 14:01:00.032405 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:00.032373    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"]
Nov 02 14:01:00.037106 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:00.037080    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367cec11-ghpfv"]
Nov 02 14:01:00.040624 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:00.040604    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"]
Nov 02 14:01:00.043480 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:00.043462    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c16d0d-xjpng"]
Nov 02 14:01:01.068781 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:01.068742    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" podUID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:01:01.541410 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:01.541325    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ce896e-9859-4665-8dc1-c330941aff19" path="/var/lib/kubelet/pods/79ce896e-9859-4665-8dc1-c330941aff19/volumes"
Nov 02 14:01:01.541844 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:01.541824    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e5a17a-a13c-43fd-825c-2d7a63c84570" path="/var/lib/kubelet/pods/96e5a17a-a13c-43fd-825c-2d7a63c84570/volumes"
Nov 02 14:01:03.018471 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.018421    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"]
Nov 02 14:01:03.021267 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.021201    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" containerID="cri-o://b75ef31780f150e1f298d99e76bc2db327d8eb68788929acc07b573a6b6ab6b9" gracePeriod=30
Nov 02 14:01:03.030992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.030953    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"]
Nov 02 14:01:03.031464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.031410    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" containerID="cri-o://782ebde05f5d121258e9507aa603563a2706e9eb0bd106f8bdc369d4fd9c7e8a" gracePeriod=30
Nov 02 14:01:03.035971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.035934    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"]
Nov 02 14:01:03.036434 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.036342    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy" containerID="cri-o://c9eea99107b0beb72c48d32618ad70e6bb1c9f76d084d2e62980e81773116a21" gracePeriod=30
Nov 02 14:01:03.847376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.847346    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 14:01:03.869609 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.869500    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:01:03.869609 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.869588    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:01:03.973225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.973197    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf4f40f-eae5-4eed-8903-7019fb0cf419-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4f40f-eae5-4eed-8903-7019fb0cf419-plugins-config-volume\") pod \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") "
Nov 02 14:01:03.973379 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.973365    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf4f40f-eae5-4eed-8903-7019fb0cf419-kube-api-access-jh2hd\" (UniqueName: \"kubernetes.io/projected/0cf4f40f-eae5-4eed-8903-7019fb0cf419-kube-api-access-jh2hd\") pod \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\" (UID: \"0cf4f40f-eae5-4eed-8903-7019fb0cf419\") "
Nov 02 14:01:03.973598 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.973544    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf4f40f-eae5-4eed-8903-7019fb0cf419-plugins-config-volume" pod "0cf4f40f-eae5-4eed-8903-7019fb0cf419" (UID: "0cf4f40f-eae5-4eed-8903-7019fb0cf419"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:03.975518 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:03.975496    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf4f40f-eae5-4eed-8903-7019fb0cf419-kube-api-access-jh2hd" pod "0cf4f40f-eae5-4eed-8903-7019fb0cf419" (UID: "0cf4f40f-eae5-4eed-8903-7019fb0cf419"). InnerVolumeSpecName "kube-api-access-jh2hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:04.074324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.074294    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2hd\" (UniqueName: \"kubernetes.io/projected/0cf4f40f-eae5-4eed-8903-7019fb0cf419-kube-api-access-jh2hd\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:04.074324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.074319    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4f40f-eae5-4eed-8903-7019fb0cf419-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:04.451211 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.451174    5572 generic.go:356] "Generic (PLEG): container finished" podID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" containerID="46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613" exitCode=0
Nov 02 14:01:04.451383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.451225    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"
Nov 02 14:01:04.451383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.451259    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" event={"ID":"0cf4f40f-eae5-4eed-8903-7019fb0cf419","Type":"ContainerDied","Data":"46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613"}
Nov 02 14:01:04.451383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.451316    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt" event={"ID":"0cf4f40f-eae5-4eed-8903-7019fb0cf419","Type":"ContainerDied","Data":"d71be863854a40ced6fc5291871f98e0fe1e0af40c5c7281bff63ffe53f56164"}
Nov 02 14:01:04.451383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.451337    5572 scope.go:122] "RemoveContainer" containerID="46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613"
Nov 02 14:01:04.461287 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.461269    5572 scope.go:122] "RemoveContainer" containerID="46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613"
Nov 02 14:01:04.461540 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:01:04.461521    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613\": container with ID starting with 46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613 not found: ID does not exist" containerID="46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613"
Nov 02 14:01:04.461735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.461571    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613"} err="failed to get container status \"46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613\": rpc error: code = NotFound desc = could not find container \"46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613\": container with ID starting with 46c1d94c5c84ddd9d989d815e58bdb4879958d6e32304710c646bde2037f6613 not found: ID does not exist"
Nov 02 14:01:04.476171 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.476145    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"]
Nov 02 14:01:04.479666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.479648    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ee7df2-84c59fbcc-2n7wt"]
Nov 02 14:01:04.569065 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.569043    5572 patch_prober.go:29] interesting pod/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:01:04.569169 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:04.569088    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:01:05.541087 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:05.541057    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" path="/var/lib/kubelet/pods/0cf4f40f-eae5-4eed-8903-7019fb0cf419/volumes"
Nov 02 14:01:06.793262 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793234    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-7pqqx"]
Nov 02 14:01:06.793676 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793662    5572 cpu_manager.go:414] "containerMap: removing container" podUID="08b957e4-a089-4e30-9750-7fcf7443a881" containerName="revoke-keys"
Nov 02 14:01:06.793719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793677    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b957e4-a089-4e30-9750-7fcf7443a881" containerName="revoke-keys"
Nov 02 14:01:06.793719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793690    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" containerName="payload-processing"
Nov 02 14:01:06.793719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793698    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" containerName="payload-processing"
Nov 02 14:01:06.793719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793717    5572 cpu_manager.go:414] "containerMap: removing container" podUID="bcc09e29-f0f5-47c7-9419-0278ceef1f7e" containerName="revoke-keys"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793722    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc09e29-f0f5-47c7-9419-0278ceef1f7e" containerName="revoke-keys"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793734    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793739    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793750    5572 cpu_manager.go:414] "containerMap: removing container" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793754    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793762    5572 cpu_manager.go:414] "containerMap: removing container" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793766    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793775    5572 cpu_manager.go:414] "containerMap: removing container" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793781    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793795    5572 cpu_manager.go:414] "containerMap: removing container" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793799    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793810    5572 cpu_manager.go:414] "containerMap: removing container" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy"
Nov 02 14:01:06.793855 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793814    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793889    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b957e4-a089-4e30-9750-7fcf7443a881" containerName="revoke-keys"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793898    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf4f40f-eae5-4eed-8903-7019fb0cf419" containerName="payload-processing"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793906    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc09e29-f0f5-47c7-9419-0278ceef1f7e" containerName="revoke-keys"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793913    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7c381f-262b-47f1-a570-1dd247e0955f" containerName="istio-proxy"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793919    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e91d94-7c4f-4d9e-ac04-04745b267604" containerName="main"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793924    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3120bf68-fa50-4b09-9725-be8ed6134529" containerName="main"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793931    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="651ba3ec-24ee-4e08-8c41-e77066d4dd96" containerName="istio-proxy"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793936    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="421a53e0-3b6f-42a5-8256-71e356f87c49" containerName="istio-proxy"
Nov 02 14:01:06.794263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.793946    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5a41fc-f27f-4ba1-9ca9-617fd1f1039c" containerName="istio-proxy"
Nov 02 14:01:06.798645 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.798627    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:06.803305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.803280    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 14:01:06.806703 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.806681    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8n6b9\""
Nov 02 14:01:06.806801 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.806762    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 14:01:06.806868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.806808    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 14:01:06.810288 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.810266    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-7pqqx"]
Nov 02 14:01:06.909196 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.909158    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtlxt\" (UniqueName: \"kubernetes.io/projected/c458a266-ea34-4c63-9ca7-1df91f87dca0-kube-api-access-xtlxt\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:06.909619 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.909585    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-metrics-certs\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:06.909808 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:06.909792    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-webhook-cert\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.011432 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.011377    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlxt\" (UniqueName: \"kubernetes.io/projected/c458a266-ea34-4c63-9ca7-1df91f87dca0-kube-api-access-xtlxt\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.011639 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.011508    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-metrics-certs\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.011639 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.011617    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-webhook-cert\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.014827 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.014797    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-webhook-cert\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.014972 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.014950    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-metrics-certs\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.026042 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.026013    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlxt\" (UniqueName: \"kubernetes.io/projected/c458a266-ea34-4c63-9ca7-1df91f87dca0-kube-api-access-xtlxt\") pod \"maas-controller-7b76f98b8f-7pqqx\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") " pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.113893 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.113866    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:07.416117 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:01:07.416079    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc458a266_ea34_4c63_9ca7_1df91f87dca0.slice/crio-5d8c4b5535684a7535cfa3e42ba028156b1fb1f0fdbd4da2be8d0c721fb959a8 WatchSource:0}: Error finding container 5d8c4b5535684a7535cfa3e42ba028156b1fb1f0fdbd4da2be8d0c721fb959a8: Status 404 returned error can't find the container with id 5d8c4b5535684a7535cfa3e42ba028156b1fb1f0fdbd4da2be8d0c721fb959a8
Nov 02 14:01:07.418770 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.418744    5572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:01:07.420970 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.420943    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-7pqqx"]
Nov 02 14:01:07.478052 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:07.478019    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx" event={"ID":"c458a266-ea34-4c63-9ca7-1df91f87dca0","Type":"ContainerStarted","Data":"5d8c4b5535684a7535cfa3e42ba028156b1fb1f0fdbd4da2be8d0c721fb959a8"}
Nov 02 14:01:08.570601 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.570469    5572 generic.go:356] "Generic (PLEG): container finished" podID="168158b6-2594-43f2-916b-9bf50573cb81" containerID="b75ef31780f150e1f298d99e76bc2db327d8eb68788929acc07b573a6b6ab6b9" exitCode=0
Nov 02 14:01:08.570601 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.570546    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" event={"ID":"168158b6-2594-43f2-916b-9bf50573cb81","Type":"ContainerDied","Data":"b75ef31780f150e1f298d99e76bc2db327d8eb68788929acc07b573a6b6ab6b9"}
Nov 02 14:01:08.572851 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.572679    5572 generic.go:356] "Generic (PLEG): container finished" podID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerID="c9eea99107b0beb72c48d32618ad70e6bb1c9f76d084d2e62980e81773116a21" exitCode=0
Nov 02 14:01:08.572851 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.572792    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" event={"ID":"81a5f392-cfbd-4849-a885-6b13b0d84781","Type":"ContainerDied","Data":"c9eea99107b0beb72c48d32618ad70e6bb1c9f76d084d2e62980e81773116a21"}
Nov 02 14:01:08.574485 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.574332    5572 generic.go:356] "Generic (PLEG): container finished" podID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerID="782ebde05f5d121258e9507aa603563a2706e9eb0bd106f8bdc369d4fd9c7e8a" exitCode=0
Nov 02 14:01:08.574485 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.574432    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" event={"ID":"92181eed-6e76-499d-8364-fd0e2dd7aa3c","Type":"ContainerDied","Data":"782ebde05f5d121258e9507aa603563a2706e9eb0bd106f8bdc369d4fd9c7e8a"}
Nov 02 14:01:08.576115 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.576099    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx" event={"ID":"c458a266-ea34-4c63-9ca7-1df91f87dca0","Type":"ContainerStarted","Data":"aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70"}
Nov 02 14:01:08.576386 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.576375    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:08.666582 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.665020    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx" podStartSLOduration=2.074259322 podStartE2EDuration="2.664999484s" totalImagesPullingTime="590.740162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:08.657674483 +0000 UTC m=+1045.235763352" watchObservedRunningTime="2026-11-02 14:01:08.664999484 +0000 UTC m=+1045.243088352"
Nov 02 14:01:08.895741 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.888729    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 14:01:08.932582 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.932127    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 14:01:08.965014 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:08.964307    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 14:01:09.054246 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.054212    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-envoy\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.054899 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.054877    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-certs\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.066961 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.066937    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-socket\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.068086 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068067    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-kube-api-access-6q6ww\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-kube-api-access-6q6ww\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.068227 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068210    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-podinfo\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.068342 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068329    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-envoy\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.068466 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068449    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-socket\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.068600 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068585    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-data\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.068740 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068725    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-certs\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.068857 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068843    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-istio-token\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-istio-token\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.068980 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.068966    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-credential-socket\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.069088 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069071    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-kube-api-access-jnmcg\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-kube-api-access-jnmcg\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.069223 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069209    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-token\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-token\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.069341 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069327    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istiod-ca-cert\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.069454 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069440    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-kube-api-access-8frrk\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-kube-api-access-8frrk\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.069584 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069550    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-credential-socket\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.069751 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069736    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-certs\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.069886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069872    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168158b6-2594-43f2-916b-9bf50573cb81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168158b6-2594-43f2-916b-9bf50573cb81-istiod-ca-cert\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.070003 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069990    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-envoy\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.070114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070101    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a5f392-cfbd-4849-a885-6b13b0d84781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a5f392-cfbd-4849-a885-6b13b0d84781-istiod-ca-cert\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.070225 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070212    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-data\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.070331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070317    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-socket\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.070444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070430    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-token\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-token\") pod \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\" (UID: \"92181eed-6e76-499d-8364-fd0e2dd7aa3c\") "
Nov 02 14:01:09.070575 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070542    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-credential-socket\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.070698 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070685    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168158b6-2594-43f2-916b-9bf50573cb81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168158b6-2594-43f2-916b-9bf50573cb81-istio-podinfo\") pod \"168158b6-2594-43f2-916b-9bf50573cb81\" (UID: \"168158b6-2594-43f2-916b-9bf50573cb81\") "
Nov 02 14:01:09.070795 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070784    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-data\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.070914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070900    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-podinfo\") pod \"81a5f392-cfbd-4849-a885-6b13b0d84781\" (UID: \"81a5f392-cfbd-4849-a885-6b13b0d84781\") "
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.065793    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-certs" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069637    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-data" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.069889    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-envoy" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.070811    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-credential-socket" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.071860    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-socket" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.071973    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-socket" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.072912    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istiod-ca-cert" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.074889    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-credential-socket" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.075061    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-data" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.075215    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-certs" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.075691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.075480    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168158b6-2594-43f2-916b-9bf50573cb81-istiod-ca-cert" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:09.076675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.076427    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-kube-api-access-8frrk" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "kube-api-access-8frrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:09.076737 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.076689    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-kube-api-access-jnmcg" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "kube-api-access-jnmcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:09.077219 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.077002    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a5f392-cfbd-4849-a885-6b13b0d84781-istiod-ca-cert" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:09.077527 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.077363    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-credential-socket" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.077959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.077692    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-socket" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.078669 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.078623    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-certs" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.078928 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.078825    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-data" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.079013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.078996    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-envoy" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.079550 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.079467    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-kube-api-access-6q6ww" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "kube-api-access-6q6ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:09.079550 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.079477    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-token" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:09.079550 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.079502    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-istio-token" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:09.079550 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.079505    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-podinfo" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:01:09.080010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.079972    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-token" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:09.081104 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.081039    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-podinfo" pod "81a5f392-cfbd-4849-a885-6b13b0d84781" (UID: "81a5f392-cfbd-4849-a885-6b13b0d84781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:01:09.110098 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.096811    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-envoy" pod "92181eed-6e76-499d-8364-fd0e2dd7aa3c" (UID: "92181eed-6e76-499d-8364-fd0e2dd7aa3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:01:09.168464 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.168417    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168158b6-2594-43f2-916b-9bf50573cb81-istio-podinfo" pod "168158b6-2594-43f2-916b-9bf50573cb81" (UID: "168158b6-2594-43f2-916b-9bf50573cb81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173242    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173274    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173292    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frrk\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-kube-api-access-8frrk\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173308    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173324    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173339    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168158b6-2594-43f2-916b-9bf50573cb81-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173352    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173367    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a5f392-cfbd-4849-a885-6b13b0d84781-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173381    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173394    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173409    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173424    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173439    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168158b6-2594-43f2-916b-9bf50573cb81-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173454    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173468    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a5f392-cfbd-4849-a885-6b13b0d84781-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173481    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173495    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173508    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173521    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6ww\" (UniqueName: \"kubernetes.io/projected/92181eed-6e76-499d-8364-fd0e2dd7aa3c-kube-api-access-6q6ww\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173534    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173548    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173580    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92181eed-6e76-499d-8364-fd0e2dd7aa3c-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173602    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173615    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168158b6-2594-43f2-916b-9bf50573cb81-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173628    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168158b6-2594-43f2-916b-9bf50573cb81-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173641    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a5f392-cfbd-4849-a885-6b13b0d84781-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.173677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.173655    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmcg\" (UniqueName: \"kubernetes.io/projected/81a5f392-cfbd-4849-a885-6b13b0d84781-kube-api-access-jnmcg\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:09.588966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.588935    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"
Nov 02 14:01:09.588966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.588939    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn" event={"ID":"168158b6-2594-43f2-916b-9bf50573cb81","Type":"ContainerDied","Data":"1fa3a528c254bbfb8e30292f82898f3393ea66d11f98814a277f358eb4dafff3"}
Nov 02 14:01:09.589463 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.589014    5572 scope.go:122] "RemoveContainer" containerID="b75ef31780f150e1f298d99e76bc2db327d8eb68788929acc07b573a6b6ab6b9"
Nov 02 14:01:09.591621 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.591600    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"
Nov 02 14:01:09.591768 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.591618    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9" event={"ID":"81a5f392-cfbd-4849-a885-6b13b0d84781","Type":"ContainerDied","Data":"f1494a46e67bc87811e6f235ba84c91329f941661d5a673c299f5e820b9bf040"}
Nov 02 14:01:09.594944 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.594925    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"
Nov 02 14:01:09.595413 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.595387    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5" event={"ID":"92181eed-6e76-499d-8364-fd0e2dd7aa3c","Type":"ContainerDied","Data":"7c1142a6ee57d477a999fdb199cce14ad032d0d6c92e14c1ed0d9c6b5ba880ae"}
Nov 02 14:01:09.601402 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.601381    5572 scope.go:122] "RemoveContainer" containerID="c9eea99107b0beb72c48d32618ad70e6bb1c9f76d084d2e62980e81773116a21"
Nov 02 14:01:09.610487 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.610466    5572 scope.go:122] "RemoveContainer" containerID="782ebde05f5d121258e9507aa603563a2706e9eb0bd106f8bdc369d4fd9c7e8a"
Nov 02 14:01:09.616191 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.616156    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"]
Nov 02 14:01:09.626210 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.626034    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-ttgqn"]
Nov 02 14:01:09.639153 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.639113    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"]
Nov 02 14:01:09.643752 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.643729    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-j9rw9"]
Nov 02 14:01:09.655936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.655903    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"]
Nov 02 14:01:09.661672 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:09.661650    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ee7df2-openshift-default-d9ffd98dc-wpjx5"]
Nov 02 14:01:11.033778 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:11.033745    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"]
Nov 02 14:01:11.037229 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:11.037203    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09847548-ss4ch"]
Nov 02 14:01:11.541653 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:11.541624    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168158b6-2594-43f2-916b-9bf50573cb81" path="/var/lib/kubelet/pods/168158b6-2594-43f2-916b-9bf50573cb81/volumes"
Nov 02 14:01:11.542052 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:11.542039    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c846bd9-df07-4ef4-a269-074f44291a34" path="/var/lib/kubelet/pods/3c846bd9-df07-4ef4-a269-074f44291a34/volumes"
Nov 02 14:01:11.542440 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:11.542427    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" path="/var/lib/kubelet/pods/81a5f392-cfbd-4849-a885-6b13b0d84781/volumes"
Nov 02 14:01:11.543120 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:11.543106    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" path="/var/lib/kubelet/pods/92181eed-6e76-499d-8364-fd0e2dd7aa3c/volumes"
Nov 02 14:01:12.025911 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:12.025877    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"]
Nov 02 14:01:12.029959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:12.029931    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"]
Nov 02 14:01:12.032729 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:12.032711    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fc5725-hgknv"]
Nov 02 14:01:12.035579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:12.035546    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fec63ed0-ksb5j"]
Nov 02 14:01:13.541385 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:13.541350    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0ec92c-8ed4-40cc-bed5-52759a164580" path="/var/lib/kubelet/pods/6f0ec92c-8ed4-40cc-bed5-52759a164580/volumes"
Nov 02 14:01:13.541798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:13.541760    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa345a17-6c1f-427c-a14e-5609e7d1292b" path="/var/lib/kubelet/pods/aa345a17-6c1f-427c-a14e-5609e7d1292b/volumes"
Nov 02 14:01:15.025600 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:15.025549    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"]
Nov 02 14:01:15.028770 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:15.028748    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e223e24-gsn46"]
Nov 02 14:01:15.541616 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:15.541580    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7" path="/var/lib/kubelet/pods/bb5d7ce5-bf2f-4036-a632-82baf1f0b9d7/volumes"
Nov 02 14:01:16.106868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:16.106836    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"]
Nov 02 14:01:16.110476 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:16.110455    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30cc84a7-k4pxd"]
Nov 02 14:01:17.541271 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:17.541239    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2c6ae6-0e92-4781-85c3-6e0b1ad40c00" path="/var/lib/kubelet/pods/de2c6ae6-0e92-4781-85c3-6e0b1ad40c00/volumes"
Nov 02 14:01:18.025620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:18.025578    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"]
Nov 02 14:01:18.029822 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:18.029792    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fffa010-q6v4p"]
Nov 02 14:01:19.025721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:19.025685    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"]
Nov 02 14:01:19.028742 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:19.028722    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1566c6af-zg69h"]
Nov 02 14:01:19.542441 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:19.542405    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58223644-0663-4482-99a6-18baa723492b" path="/var/lib/kubelet/pods/58223644-0663-4482-99a6-18baa723492b/volumes"
Nov 02 14:01:19.542867 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:19.542852    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5e08c1-af22-4809-860a-73b3a152d66a" path="/var/lib/kubelet/pods/dc5e08c1-af22-4809-860a-73b3a152d66a/volumes"
Nov 02 14:01:19.598162 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:19.598141    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:21.430379 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:01:21.430347    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:01:24.025878 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:24.025847    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"]
Nov 02 14:01:24.028812 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:24.028791    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a941dd-kz7zc"]
Nov 02 14:01:25.541191 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:25.541157    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17372a80-dcbb-4e43-8453-785c54ddbe68" path="/var/lib/kubelet/pods/17372a80-dcbb-4e43-8453-785c54ddbe68/volumes"
Nov 02 14:01:26.023953 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.023914    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"]
Nov 02 14:01:26.026837 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.026816    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-320446be-vffb6"]
Nov 02 14:01:26.743743 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.743715    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"]
Nov 02 14:01:26.744136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744107    5572 cpu_manager.go:414] "containerMap: removing container" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy"
Nov 02 14:01:26.744136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744118    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy"
Nov 02 14:01:26.744201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744150    5572 cpu_manager.go:414] "containerMap: removing container" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy"
Nov 02 14:01:26.744201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744155    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy"
Nov 02 14:01:26.744201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744163    5572 cpu_manager.go:414] "containerMap: removing container" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy"
Nov 02 14:01:26.744201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744167    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy"
Nov 02 14:01:26.744318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744243    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a5f392-cfbd-4849-a885-6b13b0d84781" containerName="istio-proxy"
Nov 02 14:01:26.744318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744256    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="92181eed-6e76-499d-8364-fd0e2dd7aa3c" containerName="istio-proxy"
Nov 02 14:01:26.744318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.744262    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="168158b6-2594-43f2-916b-9bf50573cb81" containerName="istio-proxy"
Nov 02 14:01:26.749338 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.749320    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.755255 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.755119    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:01:26.755255 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.755192    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:01:26.759952 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.759931    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"]
Nov 02 14:01:26.847596 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.847545    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.847764 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.847692    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwmn\" (UniqueName: \"kubernetes.io/projected/34482c0e-1daa-4d1f-adff-b0f3a62ba178-kube-api-access-dkwmn\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.847806 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.847772    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.948764 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.948734    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwmn\" (UniqueName: \"kubernetes.io/projected/34482c0e-1daa-4d1f-adff-b0f3a62ba178-kube-api-access-dkwmn\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.948909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.948794    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.948909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.948838    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.949497 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.949462    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.951341 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.951318    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:26.983190 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:26.983160    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwmn\" (UniqueName: \"kubernetes.io/projected/34482c0e-1daa-4d1f-adff-b0f3a62ba178-kube-api-access-dkwmn\") pod \"maas-api-revoke-sub-85237e45-fkx6b\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:27.065849 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:27.065820    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:27.349122 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:27.349095    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"]
Nov 02 14:01:27.349819 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:01:27.349786    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34482c0e_1daa_4d1f_adff_b0f3a62ba178.slice/crio-25f2036a10f8a4c3e9add80200d100edf5ee95ff4789b52d0e297e73a171f52d WatchSource:0}: Error finding container 25f2036a10f8a4c3e9add80200d100edf5ee95ff4789b52d0e297e73a171f52d: Status 404 returned error can't find the container with id 25f2036a10f8a4c3e9add80200d100edf5ee95ff4789b52d0e297e73a171f52d
Nov 02 14:01:27.541572 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:27.541527    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70905007-4f98-4865-8820-e3893d09b482" path="/var/lib/kubelet/pods/70905007-4f98-4865-8820-e3893d09b482/volumes"
Nov 02 14:01:27.724107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:27.724070    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b" event={"ID":"34482c0e-1daa-4d1f-adff-b0f3a62ba178","Type":"ContainerStarted","Data":"b19cd5e1805b1e8bae198a73aee462bdd4effc236f8f88958ba50878d4c6fcf1"}
Nov 02 14:01:27.724229 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:27.724137    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b" event={"ID":"34482c0e-1daa-4d1f-adff-b0f3a62ba178","Type":"ContainerStarted","Data":"25f2036a10f8a4c3e9add80200d100edf5ee95ff4789b52d0e297e73a171f52d"}
Nov 02 14:01:27.745789 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:27.745748    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b" podStartSLOduration=1.745734432 podStartE2EDuration="1.745734432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:27.743174562 +0000 UTC m=+1064.321263440" watchObservedRunningTime="2026-11-02 14:01:27.745734432 +0000 UTC m=+1064.323823352"
Nov 02 14:01:28.028691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:28.028617    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"]
Nov 02 14:01:28.031416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:28.031391    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b8466-pdmnc"]
Nov 02 14:01:28.034050 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:28.034021    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"]
Nov 02 14:01:28.036831 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:28.036809    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387011a-fdskf"]
Nov 02 14:01:28.733131 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:28.733097    5572 generic.go:356] "Generic (PLEG): container finished" podID="34482c0e-1daa-4d1f-adff-b0f3a62ba178" containerID="b19cd5e1805b1e8bae198a73aee462bdd4effc236f8f88958ba50878d4c6fcf1" exitCode=0
Nov 02 14:01:28.733405 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:28.733189    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b" event={"ID":"34482c0e-1daa-4d1f-adff-b0f3a62ba178","Type":"ContainerDied","Data":"b19cd5e1805b1e8bae198a73aee462bdd4effc236f8f88958ba50878d4c6fcf1"}
Nov 02 14:01:29.541274 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.541244    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aed93a7-8743-4919-a615-7a08384159b4" path="/var/lib/kubelet/pods/5aed93a7-8743-4919-a615-7a08384159b4/volumes"
Nov 02 14:01:29.541672 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.541659    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0" path="/var/lib/kubelet/pods/7317ddd1-3735-46ad-a0b9-e1e6fcccc4b0/volumes"
Nov 02 14:01:29.889282 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.889257    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:29.980086 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.980058    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-xks-ca\") pod \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") "
Nov 02 14:01:29.980276 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.980157    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34482c0e-1daa-4d1f-adff-b0f3a62ba178-kube-api-access-dkwmn\" (UniqueName: \"kubernetes.io/projected/34482c0e-1daa-4d1f-adff-b0f3a62ba178-kube-api-access-dkwmn\") pod \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") "
Nov 02 14:01:29.980276 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.980214    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-openshift-ca\") pod \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\" (UID: \"34482c0e-1daa-4d1f-adff-b0f3a62ba178\") "
Nov 02 14:01:29.980617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.980594    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-openshift-ca" pod "34482c0e-1daa-4d1f-adff-b0f3a62ba178" (UID: "34482c0e-1daa-4d1f-adff-b0f3a62ba178"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:29.982328 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.982300    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-xks-ca" pod "34482c0e-1daa-4d1f-adff-b0f3a62ba178" (UID: "34482c0e-1daa-4d1f-adff-b0f3a62ba178"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:29.982440 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:29.982422    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34482c0e-1daa-4d1f-adff-b0f3a62ba178-kube-api-access-dkwmn" pod "34482c0e-1daa-4d1f-adff-b0f3a62ba178" (UID: "34482c0e-1daa-4d1f-adff-b0f3a62ba178"). InnerVolumeSpecName "kube-api-access-dkwmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:30.081182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:30.081157    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:30.081182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:30.081181    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkwmn\" (UniqueName: \"kubernetes.io/projected/34482c0e-1daa-4d1f-adff-b0f3a62ba178-kube-api-access-dkwmn\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:30.081317 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:30.081192    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34482c0e-1daa-4d1f-adff-b0f3a62ba178-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:30.749633 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:30.749593    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"
Nov 02 14:01:30.749633 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:30.749625    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b" event={"ID":"34482c0e-1daa-4d1f-adff-b0f3a62ba178","Type":"ContainerDied","Data":"25f2036a10f8a4c3e9add80200d100edf5ee95ff4789b52d0e297e73a171f52d"}
Nov 02 14:01:30.750083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:30.749675    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f2036a10f8a4c3e9add80200d100edf5ee95ff4789b52d0e297e73a171f52d"
Nov 02 14:01:37.024507 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:37.024465    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"]
Nov 02 14:01:37.027642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:37.027616    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d4fcc-lwhsv"]
Nov 02 14:01:37.541299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:37.541264    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c655716-836b-4a8f-a0bb-850101dc5526" path="/var/lib/kubelet/pods/3c655716-836b-4a8f-a0bb-850101dc5526/volumes"
Nov 02 14:01:38.397721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.397691    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-7pqqx"]
Nov 02 14:01:38.398620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.397956    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx" podUID="c458a266-ea34-4c63-9ca7-1df91f87dca0" containerName="manager" containerID="cri-o://aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70" gracePeriod=30
Nov 02 14:01:38.644719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.644697    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:38.760918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.760849    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-webhook-cert\") pod \"c458a266-ea34-4c63-9ca7-1df91f87dca0\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") "
Nov 02 14:01:38.760918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.760909    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-metrics-certs\") pod \"c458a266-ea34-4c63-9ca7-1df91f87dca0\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") "
Nov 02 14:01:38.761086 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.761010    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c458a266-ea34-4c63-9ca7-1df91f87dca0-kube-api-access-xtlxt\" (UniqueName: \"kubernetes.io/projected/c458a266-ea34-4c63-9ca7-1df91f87dca0-kube-api-access-xtlxt\") pod \"c458a266-ea34-4c63-9ca7-1df91f87dca0\" (UID: \"c458a266-ea34-4c63-9ca7-1df91f87dca0\") "
Nov 02 14:01:38.763034 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.763004    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-webhook-cert" pod "c458a266-ea34-4c63-9ca7-1df91f87dca0" (UID: "c458a266-ea34-4c63-9ca7-1df91f87dca0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:38.763034 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.763025    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c458a266-ea34-4c63-9ca7-1df91f87dca0-kube-api-access-xtlxt" pod "c458a266-ea34-4c63-9ca7-1df91f87dca0" (UID: "c458a266-ea34-4c63-9ca7-1df91f87dca0"). InnerVolumeSpecName "kube-api-access-xtlxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:38.763197 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.763086    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-metrics-certs" pod "c458a266-ea34-4c63-9ca7-1df91f87dca0" (UID: "c458a266-ea34-4c63-9ca7-1df91f87dca0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:38.807786 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.807752    5572 generic.go:356] "Generic (PLEG): container finished" podID="c458a266-ea34-4c63-9ca7-1df91f87dca0" containerID="aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70" exitCode=0
Nov 02 14:01:38.807936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.807808    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx"
Nov 02 14:01:38.807936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.807841    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx" event={"ID":"c458a266-ea34-4c63-9ca7-1df91f87dca0","Type":"ContainerDied","Data":"aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70"}
Nov 02 14:01:38.807936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.807917    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b76f98b8f-7pqqx" event={"ID":"c458a266-ea34-4c63-9ca7-1df91f87dca0","Type":"ContainerDied","Data":"5d8c4b5535684a7535cfa3e42ba028156b1fb1f0fdbd4da2be8d0c721fb959a8"}
Nov 02 14:01:38.808063 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.807939    5572 scope.go:122] "RemoveContainer" containerID="aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70"
Nov 02 14:01:38.817963 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.817948    5572 scope.go:122] "RemoveContainer" containerID="aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70"
Nov 02 14:01:38.818249 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:01:38.818233    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70\": container with ID starting with aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70 not found: ID does not exist" containerID="aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70"
Nov 02 14:01:38.818286 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.818261    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70"} err="failed to get container status \"aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70\": rpc error: code = NotFound desc = could not find container \"aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70\": container with ID starting with aef9f9e5b71f82899e04d095f516bfd73262a03362514b4fd0a93b0e56851c70 not found: ID does not exist"
Nov 02 14:01:38.832954 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.832931    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-7pqqx"]
Nov 02 14:01:38.836921 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.836901    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-7pqqx"]
Nov 02 14:01:38.862371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.862345    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtlxt\" (UniqueName: \"kubernetes.io/projected/c458a266-ea34-4c63-9ca7-1df91f87dca0-kube-api-access-xtlxt\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:38.862371 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.862369    5572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-webhook-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:38.862490 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:38.862380    5572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c458a266-ea34-4c63-9ca7-1df91f87dca0-metrics-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:01:39.541467 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.541432    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c458a266-ea34-4c63-9ca7-1df91f87dca0" path="/var/lib/kubelet/pods/c458a266-ea34-4c63-9ca7-1df91f87dca0/volumes"
Nov 02 14:01:39.991830 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.991800    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-hmbs8"]
Nov 02 14:01:39.992187 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.992175    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c458a266-ea34-4c63-9ca7-1df91f87dca0" containerName="manager"
Nov 02 14:01:39.992232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.992188    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c458a266-ea34-4c63-9ca7-1df91f87dca0" containerName="manager"
Nov 02 14:01:39.992232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.992196    5572 cpu_manager.go:414] "containerMap: removing container" podUID="34482c0e-1daa-4d1f-adff-b0f3a62ba178" containerName="revoke-keys"
Nov 02 14:01:39.992232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.992201    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34482c0e-1daa-4d1f-adff-b0f3a62ba178" containerName="revoke-keys"
Nov 02 14:01:39.992331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.992277    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c458a266-ea34-4c63-9ca7-1df91f87dca0" containerName="manager"
Nov 02 14:01:39.992331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.992284    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="34482c0e-1daa-4d1f-adff-b0f3a62ba178" containerName="revoke-keys"
Nov 02 14:01:39.997024 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:39.997006    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.003349 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.003254    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 14:01:40.004869 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.004548    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 14:01:40.004869 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.004649    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 14:01:40.005190 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.005154    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8n6b9\""
Nov 02 14:01:40.008348 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.008329    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-hmbs8"]
Nov 02 14:01:40.027884 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.027861    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"]
Nov 02 14:01:40.031529 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.031505    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741e4fcd-ctkcn"]
Nov 02 14:01:40.174063 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.174035    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/332e62e8-9f90-4f0b-812b-f7eff688870f-webhook-cert\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.174250 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.174074    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djh67\" (UniqueName: \"kubernetes.io/projected/332e62e8-9f90-4f0b-812b-f7eff688870f-kube-api-access-djh67\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.174250 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.174231    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/332e62e8-9f90-4f0b-812b-f7eff688870f-metrics-certs\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.274977 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.274917    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/332e62e8-9f90-4f0b-812b-f7eff688870f-webhook-cert\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.274977 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.274949    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djh67\" (UniqueName: \"kubernetes.io/projected/332e62e8-9f90-4f0b-812b-f7eff688870f-kube-api-access-djh67\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.275124 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.275008    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/332e62e8-9f90-4f0b-812b-f7eff688870f-metrics-certs\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.277429 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.277412    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/332e62e8-9f90-4f0b-812b-f7eff688870f-metrics-certs\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.277526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.277469    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/332e62e8-9f90-4f0b-812b-f7eff688870f-webhook-cert\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.287054 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.287028    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djh67\" (UniqueName: \"kubernetes.io/projected/332e62e8-9f90-4f0b-812b-f7eff688870f-kube-api-access-djh67\") pod \"maas-controller-7b76f98b8f-hmbs8\" (UID: \"332e62e8-9f90-4f0b-812b-f7eff688870f\") " pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.312006 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.311982    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:40.597120 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.597096    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b76f98b8f-hmbs8"]
Nov 02 14:01:40.598452 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:01:40.598426    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332e62e8_9f90_4f0b_812b_f7eff688870f.slice/crio-9863a8ccd753d9dfb91549e038e4ee9f22bbaa3cf1e574f327d5a40069309deb WatchSource:0}: Error finding container 9863a8ccd753d9dfb91549e038e4ee9f22bbaa3cf1e574f327d5a40069309deb: Status 404 returned error can't find the container with id 9863a8ccd753d9dfb91549e038e4ee9f22bbaa3cf1e574f327d5a40069309deb
Nov 02 14:01:40.823540 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:40.823510    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b76f98b8f-hmbs8" event={"ID":"332e62e8-9f90-4f0b-812b-f7eff688870f","Type":"ContainerStarted","Data":"9863a8ccd753d9dfb91549e038e4ee9f22bbaa3cf1e574f327d5a40069309deb"}
Nov 02 14:01:41.541816 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:41.541782    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1d57ed-ae4c-40ca-82ab-504da24f1308" path="/var/lib/kubelet/pods/ba1d57ed-ae4c-40ca-82ab-504da24f1308/volumes"
Nov 02 14:01:41.832899 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:41.832863    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b76f98b8f-hmbs8" event={"ID":"332e62e8-9f90-4f0b-812b-f7eff688870f","Type":"ContainerStarted","Data":"469baa8a5b5fec7735e9f444fd83a1b76143b8466a90705d916f9ca5f92a28a2"}
Nov 02 14:01:41.833326 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:41.832988    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:41.856015 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:41.855961    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b76f98b8f-hmbs8" podStartSLOduration=2.445508318 podStartE2EDuration="2.855944019s" totalImagesPullingTime="410.435701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:41.853055459 +0000 UTC m=+1078.431144325" watchObservedRunningTime="2026-11-02 14:01:41.855944019 +0000 UTC m=+1078.434032886"
Nov 02 14:01:46.033159 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:46.033121    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"]
Nov 02 14:01:46.035935 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:46.035915    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d454f-6zffn"]
Nov 02 14:01:47.541769 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:47.541732    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cd6d80-eee2-486c-8b88-8f6c6b84f7d7" path="/var/lib/kubelet/pods/98cd6d80-eee2-486c-8b88-8f6c6b84f7d7/volumes"
Nov 02 14:01:51.429774 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:01:51.429736    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:01:52.841071 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:52.841042    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b76f98b8f-hmbs8"
Nov 02 14:01:55.022919 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:55.022881    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"]
Nov 02 14:01:55.025613 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:55.025583    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9a12b1-xm67m"]
Nov 02 14:01:55.542286 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:01:55.542253    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b54d20-df49-4c6d-ab61-5849a8932ac5" path="/var/lib/kubelet/pods/d5b54d20-df49-4c6d-ab61-5849a8932ac5/volumes"
Nov 02 14:02:01.028479 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:01.028445    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"]
Nov 02 14:02:01.031080 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:01.031048    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5455a7b-mrmgj"]
Nov 02 14:02:01.541760 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:01.541729    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0f1ebd-01d2-4ed1-914c-1657a788291d" path="/var/lib/kubelet/pods/2a0f1ebd-01d2-4ed1-914c-1657a788291d/volumes"
Nov 02 14:02:09.065781 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:09.065743    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"]
Nov 02 14:02:09.068314 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:09.068294    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8057c-6txtr"]
Nov 02 14:02:09.541759 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:09.541679    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa2232b-97d8-4bc3-8a37-48c3c8043569" path="/var/lib/kubelet/pods/efa2232b-97d8-4bc3-8a37-48c3c8043569/volumes"
Nov 02 14:02:16.038815 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:16.038787    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"]
Nov 02 14:02:16.041419 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:16.041400    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662b0641-jqx57"]
Nov 02 14:02:17.541658 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:17.541625    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1656d28-5b08-4f2f-ba3d-8008c6c103bf" path="/var/lib/kubelet/pods/c1656d28-5b08-4f2f-ba3d-8008c6c103bf/volumes"
Nov 02 14:02:21.436727 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:02:21.436677    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:02:34.782805 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.782772    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"]
Nov 02 14:02:34.786116 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.786095    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.802684 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.802656    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:02:34.803261 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.803235    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:02:34.804971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.804944    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"]
Nov 02 14:02:34.887611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.887580    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vmr\" (UniqueName: \"kubernetes.io/projected/497e573e-ce57-4466-bf56-fb247d3c4efd-kube-api-access-p6vmr\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.887750 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.887628    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.887750 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.887728    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.988734 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.988697    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vmr\" (UniqueName: \"kubernetes.io/projected/497e573e-ce57-4466-bf56-fb247d3c4efd-kube-api-access-p6vmr\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.988915 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.988753    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.988915 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.988810    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.989416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.989391    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:34.991268 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:34.991247    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:35.001940 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:35.001913    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vmr\" (UniqueName: \"kubernetes.io/projected/497e573e-ce57-4466-bf56-fb247d3c4efd-kube-api-access-p6vmr\") pod \"maas-api-revoke-sub-c1e271a9-d28vk\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:35.106610 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:35.106588    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:35.390363 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:35.390330    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"]
Nov 02 14:02:35.391256 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:02:35.391231    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497e573e_ce57_4466_bf56_fb247d3c4efd.slice/crio-f204182f246cba360f71b0a1d03f4888eebbacc14e7902f2f609c1bcb7a60e35 WatchSource:0}: Error finding container f204182f246cba360f71b0a1d03f4888eebbacc14e7902f2f609c1bcb7a60e35: Status 404 returned error can't find the container with id f204182f246cba360f71b0a1d03f4888eebbacc14e7902f2f609c1bcb7a60e35
Nov 02 14:02:36.239941 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:36.239908    5572 generic.go:356] "Generic (PLEG): container finished" podID="497e573e-ce57-4466-bf56-fb247d3c4efd" containerID="d9ac2b07804fe566979bdbfb9c9d9f8113dfcb4f9e153b7d5bc92fc350808da9" exitCode=0
Nov 02 14:02:36.240279 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:36.239956    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk" event={"ID":"497e573e-ce57-4466-bf56-fb247d3c4efd","Type":"ContainerDied","Data":"d9ac2b07804fe566979bdbfb9c9d9f8113dfcb4f9e153b7d5bc92fc350808da9"}
Nov 02 14:02:36.240279 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:36.240020    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk" event={"ID":"497e573e-ce57-4466-bf56-fb247d3c4efd","Type":"ContainerStarted","Data":"f204182f246cba360f71b0a1d03f4888eebbacc14e7902f2f609c1bcb7a60e35"}
Nov 02 14:02:37.384879 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.384857    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:37.513983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.513906    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497e573e-ce57-4466-bf56-fb247d3c4efd-kube-api-access-p6vmr\" (UniqueName: \"kubernetes.io/projected/497e573e-ce57-4466-bf56-fb247d3c4efd-kube-api-access-p6vmr\") pod \"497e573e-ce57-4466-bf56-fb247d3c4efd\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") "
Nov 02 14:02:37.513983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.513979    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-openshift-ca\") pod \"497e573e-ce57-4466-bf56-fb247d3c4efd\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") "
Nov 02 14:02:37.514192 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.514045    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-xks-ca\") pod \"497e573e-ce57-4466-bf56-fb247d3c4efd\" (UID: \"497e573e-ce57-4466-bf56-fb247d3c4efd\") "
Nov 02 14:02:37.514391 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.514359    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-openshift-ca" pod "497e573e-ce57-4466-bf56-fb247d3c4efd" (UID: "497e573e-ce57-4466-bf56-fb247d3c4efd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:02:37.516128 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.516107    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-xks-ca" pod "497e573e-ce57-4466-bf56-fb247d3c4efd" (UID: "497e573e-ce57-4466-bf56-fb247d3c4efd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:02:37.516128 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.516111    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497e573e-ce57-4466-bf56-fb247d3c4efd-kube-api-access-p6vmr" pod "497e573e-ce57-4466-bf56-fb247d3c4efd" (UID: "497e573e-ce57-4466-bf56-fb247d3c4efd"). InnerVolumeSpecName "kube-api-access-p6vmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:02:37.615217 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.615186    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:02:37.615217 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.615214    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497e573e-ce57-4466-bf56-fb247d3c4efd-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:02:37.615394 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:37.615224    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vmr\" (UniqueName: \"kubernetes.io/projected/497e573e-ce57-4466-bf56-fb247d3c4efd-kube-api-access-p6vmr\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:02:38.256155 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:38.256126    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"
Nov 02 14:02:38.256360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:38.256158    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk" event={"ID":"497e573e-ce57-4466-bf56-fb247d3c4efd","Type":"ContainerDied","Data":"f204182f246cba360f71b0a1d03f4888eebbacc14e7902f2f609c1bcb7a60e35"}
Nov 02 14:02:38.256360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:38.256207    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f204182f246cba360f71b0a1d03f4888eebbacc14e7902f2f609c1bcb7a60e35"
Nov 02 14:02:44.057501 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.057462    5572 scope.go:122] "RemoveContainer" containerID="c027c665d57e5f0d30b2fc8307e3b12e44e577d7e10a806af7a8bf1952755321"
Nov 02 14:02:44.067348 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.067326    5572 scope.go:122] "RemoveContainer" containerID="8ac3796dcdc6f4e2c53560612e5d7f02cff3ece6a39fd506aa98807545088c7e"
Nov 02 14:02:44.076698 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.076679    5572 scope.go:122] "RemoveContainer" containerID="cea8fee5ac7c07b3e137ad8cf721592559051350bf96eebcb1b7ec459a5546de"
Nov 02 14:02:44.086143 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.086118    5572 scope.go:122] "RemoveContainer" containerID="0b7737786343130063824addcff5cd308e2a0badcc9f2dc214d264a9166cf9eb"
Nov 02 14:02:44.098546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.098515    5572 scope.go:122] "RemoveContainer" containerID="52d50a850a2d624b8e956c7f3bfaf11c2b67b0b020e4b84c53cdf7014a0dd5f2"
Nov 02 14:02:44.108149 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.108128    5572 scope.go:122] "RemoveContainer" containerID="923efe050c74fe4b671065a832d059bf9045356174b50dbd4ddddba9c45234f0"
Nov 02 14:02:44.117172 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.117152    5572 scope.go:122] "RemoveContainer" containerID="87ac09802ea30ff73a6b1651ac6df8fa86369fc309d65e0e19c510d9e64de4d3"
Nov 02 14:02:44.126161 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.126138    5572 scope.go:122] "RemoveContainer" containerID="ce36cb3cb5ae0be452325f742298d44cd188b573218542c2f89a5b3480beeae7"
Nov 02 14:02:44.134946 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.134928    5572 scope.go:122] "RemoveContainer" containerID="644256b1f9c2ab16ce03ba83c3f3e1a8186b42cd276aa4b7cb06ce5d0262dce5"
Nov 02 14:02:44.143775 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.143754    5572 scope.go:122] "RemoveContainer" containerID="03673cb442edeb25efba260484a2ed7b646dae059fa5f4d6a8d9b8a227c9cdcf"
Nov 02 14:02:44.152422 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.152402    5572 scope.go:122] "RemoveContainer" containerID="ca50b15efbeaad7b708207ba47cb3e6fa69c32ef785f37e22812af638b4997f2"
Nov 02 14:02:44.161219 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.161197    5572 scope.go:122] "RemoveContainer" containerID="dcf2aeb4bcf016d6b25e9ce269436cc1b6478af6f5d72003892ca4b68b3751b3"
Nov 02 14:02:44.169576 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.169529    5572 scope.go:122] "RemoveContainer" containerID="4f12981cad13470917919216da973d2777f38c960e47a8636342b4c7399d9459"
Nov 02 14:02:44.177854 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.177834    5572 scope.go:122] "RemoveContainer" containerID="55b2533454401fe19cb30781842b2f686eddf8d03a1f509ffebd91db5b0dd15f"
Nov 02 14:02:44.186811 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.186788    5572 scope.go:122] "RemoveContainer" containerID="c91a8dd257bc5ad8c6f7742f9d972a33ff477e3962dc00158d9dc77cdb7b10aa"
Nov 02 14:02:44.195806 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.195786    5572 scope.go:122] "RemoveContainer" containerID="1cb30395c80775332042afb0370a1a99cd6ff5b38c46c8aabe2233fe4be6b727"
Nov 02 14:02:44.204671 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.204649    5572 scope.go:122] "RemoveContainer" containerID="ea8e2509ba6c27b28de48a4b355665bbbf67b11122292a43676500d6543c6502"
Nov 02 14:02:44.213421 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.213400    5572 scope.go:122] "RemoveContainer" containerID="67a66b4eceef5c31b6dcab51155e99c6bdccadf2d265683eee1f5e2c4f25a5ae"
Nov 02 14:02:44.222411 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:44.222393    5572 scope.go:122] "RemoveContainer" containerID="6cec2ba3200f535391b4d25583c9d4fabfd86e598553fb1393df5b7f609865e7"
Nov 02 14:02:51.430070 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:02:51.430034    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:02:54.024377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:54.024335    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"]
Nov 02 14:02:54.027172 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:54.027144    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64ed6a7-8n5mw"]
Nov 02 14:02:55.541436 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.541406    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3905545-6743-4511-9d4c-8fc0bacaafdc" path="/var/lib/kubelet/pods/e3905545-6743-4511-9d4c-8fc0bacaafdc/volumes"
Nov 02 14:02:55.645592 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.645541    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"]
Nov 02 14:02:55.646134 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.646118    5572 cpu_manager.go:414] "containerMap: removing container" podUID="497e573e-ce57-4466-bf56-fb247d3c4efd" containerName="revoke-keys"
Nov 02 14:02:55.646181 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.646136    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497e573e-ce57-4466-bf56-fb247d3c4efd" containerName="revoke-keys"
Nov 02 14:02:55.646280 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.646269    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="497e573e-ce57-4466-bf56-fb247d3c4efd" containerName="revoke-keys"
Nov 02 14:02:55.653038 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.653009    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.656859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.656831    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"]
Nov 02 14:02:55.657228 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.657206    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:02:55.658252 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.658234    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:02:55.709700 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.709666    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.709826 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.709707    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2r8\" (UniqueName: \"kubernetes.io/projected/4efe2a3b-d746-4290-a243-d305bb3ac01c-kube-api-access-cc2r8\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.709826 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.709778    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.810376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.810349    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.810537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.810390    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2r8\" (UniqueName: \"kubernetes.io/projected/4efe2a3b-d746-4290-a243-d305bb3ac01c-kube-api-access-cc2r8\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.810668 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.810643    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.811005 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.810987    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.813097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.813070    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.823121 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.823102    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2r8\" (UniqueName: \"kubernetes.io/projected/4efe2a3b-d746-4290-a243-d305bb3ac01c-kube-api-access-cc2r8\") pod \"maas-api-revoke-sub-2e1f5b5f-pt46q\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:55.968688 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:55.968668    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:56.247358 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:56.247329    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"]
Nov 02 14:02:56.249000 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:02:56.248978    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe2a3b_d746_4290_a243_d305bb3ac01c.slice/crio-fbaabd1998c68eb3b7b22d5ef90c1ba4e0fd054ca42c7072dcccf8a7c0eb5f3e WatchSource:0}: Error finding container fbaabd1998c68eb3b7b22d5ef90c1ba4e0fd054ca42c7072dcccf8a7c0eb5f3e: Status 404 returned error can't find the container with id fbaabd1998c68eb3b7b22d5ef90c1ba4e0fd054ca42c7072dcccf8a7c0eb5f3e
Nov 02 14:02:56.397060 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:56.397031    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q" event={"ID":"4efe2a3b-d746-4290-a243-d305bb3ac01c","Type":"ContainerStarted","Data":"fbaabd1998c68eb3b7b22d5ef90c1ba4e0fd054ca42c7072dcccf8a7c0eb5f3e"}
Nov 02 14:02:57.404488 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:57.404397    5572 generic.go:356] "Generic (PLEG): container finished" podID="4efe2a3b-d746-4290-a243-d305bb3ac01c" containerID="24a7fd2f6230328cd6199833eb6d18afc4043a1fee07fb4f1d045fc43383586a" exitCode=0
Nov 02 14:02:57.404940 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:57.404487    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q" event={"ID":"4efe2a3b-d746-4290-a243-d305bb3ac01c","Type":"ContainerDied","Data":"24a7fd2f6230328cd6199833eb6d18afc4043a1fee07fb4f1d045fc43383586a"}
Nov 02 14:02:58.548166 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.548142    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:58.633421 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.633383    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-xks-ca\") pod \"4efe2a3b-d746-4290-a243-d305bb3ac01c\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") "
Nov 02 14:02:58.633622 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.633494    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-openshift-ca\") pod \"4efe2a3b-d746-4290-a243-d305bb3ac01c\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") "
Nov 02 14:02:58.633622 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.633607    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efe2a3b-d746-4290-a243-d305bb3ac01c-kube-api-access-cc2r8\" (UniqueName: \"kubernetes.io/projected/4efe2a3b-d746-4290-a243-d305bb3ac01c-kube-api-access-cc2r8\") pod \"4efe2a3b-d746-4290-a243-d305bb3ac01c\" (UID: \"4efe2a3b-d746-4290-a243-d305bb3ac01c\") "
Nov 02 14:02:58.633886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.633858    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-openshift-ca" pod "4efe2a3b-d746-4290-a243-d305bb3ac01c" (UID: "4efe2a3b-d746-4290-a243-d305bb3ac01c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:02:58.634022 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.633979    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:02:58.635714 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.635692    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-xks-ca" pod "4efe2a3b-d746-4290-a243-d305bb3ac01c" (UID: "4efe2a3b-d746-4290-a243-d305bb3ac01c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:02:58.635844 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.635823    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe2a3b-d746-4290-a243-d305bb3ac01c-kube-api-access-cc2r8" pod "4efe2a3b-d746-4290-a243-d305bb3ac01c" (UID: "4efe2a3b-d746-4290-a243-d305bb3ac01c"). InnerVolumeSpecName "kube-api-access-cc2r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:02:58.734891 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.734823    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2r8\" (UniqueName: \"kubernetes.io/projected/4efe2a3b-d746-4290-a243-d305bb3ac01c-kube-api-access-cc2r8\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:02:58.734891 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:58.734851    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efe2a3b-d746-4290-a243-d305bb3ac01c-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:02:59.417659 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:59.417632    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"
Nov 02 14:02:59.417825 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:59.417649    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q" event={"ID":"4efe2a3b-d746-4290-a243-d305bb3ac01c","Type":"ContainerDied","Data":"fbaabd1998c68eb3b7b22d5ef90c1ba4e0fd054ca42c7072dcccf8a7c0eb5f3e"}
Nov 02 14:02:59.417825 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:02:59.417708    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbaabd1998c68eb3b7b22d5ef90c1ba4e0fd054ca42c7072dcccf8a7c0eb5f3e"
Nov 02 14:03:06.521414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.521380    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"]
Nov 02 14:03:06.521973 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.521810    5572 cpu_manager.go:414] "containerMap: removing container" podUID="4efe2a3b-d746-4290-a243-d305bb3ac01c" containerName="revoke-keys"
Nov 02 14:03:06.521973 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.521821    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe2a3b-d746-4290-a243-d305bb3ac01c" containerName="revoke-keys"
Nov 02 14:03:06.521973 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.521899    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe2a3b-d746-4290-a243-d305bb3ac01c" containerName="revoke-keys"
Nov 02 14:03:06.525263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.525247    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.531041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.531004    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:03:06.532763 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.532326    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:03:06.537104 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.537079    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"]
Nov 02 14:03:06.607288 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.607257    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.607479 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.607314    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.607479 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.607435    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97kf\" (UniqueName: \"kubernetes.io/projected/fe0d94a4-78f3-4395-8b49-cc810e95a407-kube-api-access-g97kf\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.708104 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.708072    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.708269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.708139    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.708269 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.708180    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97kf\" (UniqueName: \"kubernetes.io/projected/fe0d94a4-78f3-4395-8b49-cc810e95a407-kube-api-access-g97kf\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.708750 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.708728    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.710670 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.710652    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.721455 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.721435    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97kf\" (UniqueName: \"kubernetes.io/projected/fe0d94a4-78f3-4395-8b49-cc810e95a407-kube-api-access-g97kf\") pod \"maas-api-revoke-sub-ec843018-s97kd\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:06.843897 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:06.843862    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:07.138349 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:07.138323    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"]
Nov 02 14:03:07.139228 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:03:07.139203    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0d94a4_78f3_4395_8b49_cc810e95a407.slice/crio-9e76ff4efb75d9ff6795b9d45822acfc4c576d7ac4dbf8cb9b0c81886b458491 WatchSource:0}: Error finding container 9e76ff4efb75d9ff6795b9d45822acfc4c576d7ac4dbf8cb9b0c81886b458491: Status 404 returned error can't find the container with id 9e76ff4efb75d9ff6795b9d45822acfc4c576d7ac4dbf8cb9b0c81886b458491
Nov 02 14:03:07.468725 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:07.468683    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd" event={"ID":"fe0d94a4-78f3-4395-8b49-cc810e95a407","Type":"ContainerStarted","Data":"a03688161ba04d865ebb0caff9c6bac4e50741bbab22d63ecdd9941b60a7b6f1"}
Nov 02 14:03:07.468852 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:07.468746    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd" event={"ID":"fe0d94a4-78f3-4395-8b49-cc810e95a407","Type":"ContainerStarted","Data":"9e76ff4efb75d9ff6795b9d45822acfc4c576d7ac4dbf8cb9b0c81886b458491"}
Nov 02 14:03:07.497538 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:07.497489    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd" podStartSLOduration=1.497475183 podStartE2EDuration="1.497475183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:07.495385716 +0000 UTC m=+1164.073474582" watchObservedRunningTime="2026-11-02 14:03:07.497475183 +0000 UTC m=+1164.075564049"
Nov 02 14:03:08.476323 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:08.476288    5572 generic.go:356] "Generic (PLEG): container finished" podID="fe0d94a4-78f3-4395-8b49-cc810e95a407" containerID="a03688161ba04d865ebb0caff9c6bac4e50741bbab22d63ecdd9941b60a7b6f1" exitCode=0
Nov 02 14:03:08.476736 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:08.476340    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd" event={"ID":"fe0d94a4-78f3-4395-8b49-cc810e95a407","Type":"ContainerDied","Data":"a03688161ba04d865ebb0caff9c6bac4e50741bbab22d63ecdd9941b60a7b6f1"}
Nov 02 14:03:09.024834 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.024799    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"]
Nov 02 14:03:09.030027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.030004    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97c4f20f-4n8tl"]
Nov 02 14:03:09.545715 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.545676    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1078fd-5ea3-4888-ac87-787f791cc6dd" path="/var/lib/kubelet/pods/2e1078fd-5ea3-4888-ac87-787f791cc6dd/volumes"
Nov 02 14:03:09.655444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.655416    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:09.743156 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.743067    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0d94a4-78f3-4395-8b49-cc810e95a407-kube-api-access-g97kf\" (UniqueName: \"kubernetes.io/projected/fe0d94a4-78f3-4395-8b49-cc810e95a407-kube-api-access-g97kf\") pod \"fe0d94a4-78f3-4395-8b49-cc810e95a407\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") "
Nov 02 14:03:09.743156 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.743148    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-openshift-ca\") pod \"fe0d94a4-78f3-4395-8b49-cc810e95a407\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") "
Nov 02 14:03:09.743389 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.743204    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-xks-ca\") pod \"fe0d94a4-78f3-4395-8b49-cc810e95a407\" (UID: \"fe0d94a4-78f3-4395-8b49-cc810e95a407\") "
Nov 02 14:03:09.743578 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.743529    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-openshift-ca" pod "fe0d94a4-78f3-4395-8b49-cc810e95a407" (UID: "fe0d94a4-78f3-4395-8b49-cc810e95a407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:09.745396 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.745373    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0d94a4-78f3-4395-8b49-cc810e95a407-kube-api-access-g97kf" pod "fe0d94a4-78f3-4395-8b49-cc810e95a407" (UID: "fe0d94a4-78f3-4395-8b49-cc810e95a407"). InnerVolumeSpecName "kube-api-access-g97kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:09.745858 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.745838    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-xks-ca" pod "fe0d94a4-78f3-4395-8b49-cc810e95a407" (UID: "fe0d94a4-78f3-4395-8b49-cc810e95a407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:09.844364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.844321    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:09.844364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.844360    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0d94a4-78f3-4395-8b49-cc810e95a407-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:09.844611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:09.844377    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g97kf\" (UniqueName: \"kubernetes.io/projected/fe0d94a4-78f3-4395-8b49-cc810e95a407-kube-api-access-g97kf\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:10.521585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:10.520927    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd" event={"ID":"fe0d94a4-78f3-4395-8b49-cc810e95a407","Type":"ContainerDied","Data":"9e76ff4efb75d9ff6795b9d45822acfc4c576d7ac4dbf8cb9b0c81886b458491"}
Nov 02 14:03:10.521585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:10.521030    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e76ff4efb75d9ff6795b9d45822acfc4c576d7ac4dbf8cb9b0c81886b458491"
Nov 02 14:03:10.521585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:10.521103    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"
Nov 02 14:03:21.051860 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:21.051744    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"]
Nov 02 14:03:21.055787 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:21.055758    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d90261-r4q72"]
Nov 02 14:03:21.475799 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:03:21.475762    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:03:21.559634 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:21.559240    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810bd192-0081-4082-9408-2eda4b79727d" path="/var/lib/kubelet/pods/810bd192-0081-4082-9408-2eda4b79727d/volumes"
Nov 02 14:03:24.023914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:24.023826    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"]
Nov 02 14:03:24.027186 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:24.027164    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b0e8b9-wzwkk"]
Nov 02 14:03:25.541415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:25.541381    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe3461a-0791-4ed6-b2f7-e07b5c01299d" path="/var/lib/kubelet/pods/8fe3461a-0791-4ed6-b2f7-e07b5c01299d/volumes"
Nov 02 14:03:32.900111 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.900077    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"]
Nov 02 14:03:32.900546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.900532    5572 cpu_manager.go:414] "containerMap: removing container" podUID="fe0d94a4-78f3-4395-8b49-cc810e95a407" containerName="revoke-keys"
Nov 02 14:03:32.900628 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.900548    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0d94a4-78f3-4395-8b49-cc810e95a407" containerName="revoke-keys"
Nov 02 14:03:32.900716 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.900704    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0d94a4-78f3-4395-8b49-cc810e95a407" containerName="revoke-keys"
Nov 02 14:03:32.903900 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.903879    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:32.911737 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.911714    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:03:32.911868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.911839    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:03:32.915255 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.915231    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"]
Nov 02 14:03:32.996606 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.996579    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:32.996606 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.996613    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffqf\" (UniqueName: \"kubernetes.io/projected/67eb721a-16f5-4d13-b219-23c56707ad93-kube-api-access-2ffqf\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:32.996774 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:32.996637    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.097376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.097350    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.097505 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.097383    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffqf\" (UniqueName: \"kubernetes.io/projected/67eb721a-16f5-4d13-b219-23c56707ad93-kube-api-access-2ffqf\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.097505 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.097406    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.097988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.097970    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.099760 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.099743    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.110289 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.110263    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffqf\" (UniqueName: \"kubernetes.io/projected/67eb721a-16f5-4d13-b219-23c56707ad93-kube-api-access-2ffqf\") pod \"maas-api-revoke-sub-046e1343-5rllj\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.218491 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.218421    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:33.499324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.499291    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"]
Nov 02 14:03:33.500418 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:03:33.500394    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67eb721a_16f5_4d13_b219_23c56707ad93.slice/crio-5faee3be70ce825b97cf96130a90b5a7348b55a0d3cb5d8705617db5a04ebf33 WatchSource:0}: Error finding container 5faee3be70ce825b97cf96130a90b5a7348b55a0d3cb5d8705617db5a04ebf33: Status 404 returned error can't find the container with id 5faee3be70ce825b97cf96130a90b5a7348b55a0d3cb5d8705617db5a04ebf33
Nov 02 14:03:33.700345 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:33.700310    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj" event={"ID":"67eb721a-16f5-4d13-b219-23c56707ad93","Type":"ContainerStarted","Data":"5faee3be70ce825b97cf96130a90b5a7348b55a0d3cb5d8705617db5a04ebf33"}
Nov 02 14:03:34.708206 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:34.708175    5572 generic.go:356] "Generic (PLEG): container finished" podID="67eb721a-16f5-4d13-b219-23c56707ad93" containerID="1d670896a930ccdf41c0ee800945ad24ac47b8ed7560672f2c94a2f9006eda62" exitCode=0
Nov 02 14:03:34.708598 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:34.708255    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj" event={"ID":"67eb721a-16f5-4d13-b219-23c56707ad93","Type":"ContainerDied","Data":"1d670896a930ccdf41c0ee800945ad24ac47b8ed7560672f2c94a2f9006eda62"}
Nov 02 14:03:35.851931 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:35.851904    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:35.922934 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:35.922906    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-xks-ca\") pod \"67eb721a-16f5-4d13-b219-23c56707ad93\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") "
Nov 02 14:03:35.923108 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:35.922962    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-openshift-ca\") pod \"67eb721a-16f5-4d13-b219-23c56707ad93\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") "
Nov 02 14:03:35.923108 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:35.923070    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67eb721a-16f5-4d13-b219-23c56707ad93-kube-api-access-2ffqf\" (UniqueName: \"kubernetes.io/projected/67eb721a-16f5-4d13-b219-23c56707ad93-kube-api-access-2ffqf\") pod \"67eb721a-16f5-4d13-b219-23c56707ad93\" (UID: \"67eb721a-16f5-4d13-b219-23c56707ad93\") "
Nov 02 14:03:35.923333 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:35.923309    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-openshift-ca" pod "67eb721a-16f5-4d13-b219-23c56707ad93" (UID: "67eb721a-16f5-4d13-b219-23c56707ad93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:35.925200 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:35.925174    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-xks-ca" pod "67eb721a-16f5-4d13-b219-23c56707ad93" (UID: "67eb721a-16f5-4d13-b219-23c56707ad93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:35.925379 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:35.925354    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67eb721a-16f5-4d13-b219-23c56707ad93-kube-api-access-2ffqf" pod "67eb721a-16f5-4d13-b219-23c56707ad93" (UID: "67eb721a-16f5-4d13-b219-23c56707ad93"). InnerVolumeSpecName "kube-api-access-2ffqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:36.023778 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.023723    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:36.023778 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.023746    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffqf\" (UniqueName: \"kubernetes.io/projected/67eb721a-16f5-4d13-b219-23c56707ad93-kube-api-access-2ffqf\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:36.023778 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.023756    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eb721a-16f5-4d13-b219-23c56707ad93-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:36.099813 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.099782    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"]
Nov 02 14:03:36.107749 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.107723    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-397ec389-jwnw5"]
Nov 02 14:03:36.721611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.721576    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"
Nov 02 14:03:36.721811 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.721619    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj" event={"ID":"67eb721a-16f5-4d13-b219-23c56707ad93","Type":"ContainerDied","Data":"5faee3be70ce825b97cf96130a90b5a7348b55a0d3cb5d8705617db5a04ebf33"}
Nov 02 14:03:36.721811 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:36.721669    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5faee3be70ce825b97cf96130a90b5a7348b55a0d3cb5d8705617db5a04ebf33"
Nov 02 14:03:37.542197 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:37.542164    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47c2db8-393e-450f-8dad-28e3cddb860d" path="/var/lib/kubelet/pods/a47c2db8-393e-450f-8dad-28e3cddb860d/volumes"
Nov 02 14:03:39.033022 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:39.032994    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"]
Nov 02 14:03:39.037257 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:39.037229    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb51c58e-9pvsf"]
Nov 02 14:03:39.541572 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:39.541537    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd49051-1365-4f49-83bd-26765a94eaac" path="/var/lib/kubelet/pods/ccd49051-1365-4f49-83bd-26765a94eaac/volumes"
Nov 02 14:03:43.115796 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.115754    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"]
Nov 02 14:03:43.116190 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.116158    5572 cpu_manager.go:414] "containerMap: removing container" podUID="67eb721a-16f5-4d13-b219-23c56707ad93" containerName="revoke-keys"
Nov 02 14:03:43.116190 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.116169    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67eb721a-16f5-4d13-b219-23c56707ad93" containerName="revoke-keys"
Nov 02 14:03:43.116265 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.116244    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="67eb721a-16f5-4d13-b219-23c56707ad93" containerName="revoke-keys"
Nov 02 14:03:43.120805 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.120787    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.128850 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.128823    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:03:43.128972 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.128897    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:03:43.132072 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.132049    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"]
Nov 02 14:03:43.295392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.295353    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.295629 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.295478    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbjm\" (UniqueName: \"kubernetes.io/projected/c134796e-9f06-4068-b8a8-875673cb5570-kube-api-access-5bbjm\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.295629 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.295501    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.396854 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.396771    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.396854 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.396836    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbjm\" (UniqueName: \"kubernetes.io/projected/c134796e-9f06-4068-b8a8-875673cb5570-kube-api-access-5bbjm\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.397041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.396858    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.397500 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.397477    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.399317 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.399300    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.411474 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.411448    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbjm\" (UniqueName: \"kubernetes.io/projected/c134796e-9f06-4068-b8a8-875673cb5570-kube-api-access-5bbjm\") pod \"maas-api-revoke-sub-d3fba056-tzvjx\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.435476 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.435452    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:43.712721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.712694    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"]
Nov 02 14:03:43.714099 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:03:43.714077    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc134796e_9f06_4068_b8a8_875673cb5570.slice/crio-5f8321b1f49a4a2a6982b8d484b3ad59f32d251b1031dbc6cbde39e216b95a0b WatchSource:0}: Error finding container 5f8321b1f49a4a2a6982b8d484b3ad59f32d251b1031dbc6cbde39e216b95a0b: Status 404 returned error can't find the container with id 5f8321b1f49a4a2a6982b8d484b3ad59f32d251b1031dbc6cbde39e216b95a0b
Nov 02 14:03:43.766434 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:43.766399    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx" event={"ID":"c134796e-9f06-4068-b8a8-875673cb5570","Type":"ContainerStarted","Data":"5f8321b1f49a4a2a6982b8d484b3ad59f32d251b1031dbc6cbde39e216b95a0b"}
Nov 02 14:03:44.340514 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:44.340465    5572 scope.go:122] "RemoveContainer" containerID="bcd45432acc2042531613f19037abb4a8713fcd9e5ce75efc4f525adee5407b1"
Nov 02 14:03:44.352885 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:44.352857    5572 scope.go:122] "RemoveContainer" containerID="00dd08e5d2114c389e19cc6874aebf5d8f69a78e89569754c0c1d6df547e8318"
Nov 02 14:03:44.363939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:44.363916    5572 scope.go:122] "RemoveContainer" containerID="c37a91ef9b9797f6cd0b08ae362c00ca0f4c2e5564c42a1ab31e7ba9353b80ed"
Nov 02 14:03:44.374032 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:44.374006    5572 scope.go:122] "RemoveContainer" containerID="00b78bf55b98ed205b68d6cc4ef41188d69ec55b73401aa902ae5d883fe29382"
Nov 02 14:03:44.385388 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:44.385359    5572 scope.go:122] "RemoveContainer" containerID="bb718e678cab0068a58dc31e681339c857ff1ee566e2b93e7a259d53c958bd24"
Nov 02 14:03:44.774801 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:44.774766    5572 generic.go:356] "Generic (PLEG): container finished" podID="c134796e-9f06-4068-b8a8-875673cb5570" containerID="020eb00c0e0d1d11f0b571c6ac93084b558a2ac82f6f8739226b1ff4cb2e4ae1" exitCode=0
Nov 02 14:03:44.775004 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:44.774852    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx" event={"ID":"c134796e-9f06-4068-b8a8-875673cb5570","Type":"ContainerDied","Data":"020eb00c0e0d1d11f0b571c6ac93084b558a2ac82f6f8739226b1ff4cb2e4ae1"}
Nov 02 14:03:46.333499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.333144    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:46.452360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.452141    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c134796e-9f06-4068-b8a8-875673cb5570-kube-api-access-5bbjm\" (UniqueName: \"kubernetes.io/projected/c134796e-9f06-4068-b8a8-875673cb5570-kube-api-access-5bbjm\") pod \"c134796e-9f06-4068-b8a8-875673cb5570\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") "
Nov 02 14:03:46.453191 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.453156    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-openshift-ca\") pod \"c134796e-9f06-4068-b8a8-875673cb5570\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") "
Nov 02 14:03:46.453330 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.453248    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-xks-ca\") pod \"c134796e-9f06-4068-b8a8-875673cb5570\" (UID: \"c134796e-9f06-4068-b8a8-875673cb5570\") "
Nov 02 14:03:46.454191 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.454150    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-openshift-ca" pod "c134796e-9f06-4068-b8a8-875673cb5570" (UID: "c134796e-9f06-4068-b8a8-875673cb5570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:46.467803 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.467754    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c134796e-9f06-4068-b8a8-875673cb5570-kube-api-access-5bbjm" pod "c134796e-9f06-4068-b8a8-875673cb5570" (UID: "c134796e-9f06-4068-b8a8-875673cb5570"). InnerVolumeSpecName "kube-api-access-5bbjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:46.468722 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.468701    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-xks-ca" pod "c134796e-9f06-4068-b8a8-875673cb5570" (UID: "c134796e-9f06-4068-b8a8-875673cb5570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:46.554689 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.554117    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:46.554689 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.554152    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c134796e-9f06-4068-b8a8-875673cb5570-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:46.554689 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.554162    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbjm\" (UniqueName: \"kubernetes.io/projected/c134796e-9f06-4068-b8a8-875673cb5570-kube-api-access-5bbjm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:03:46.839826 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.839797    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"
Nov 02 14:03:46.840013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.839839    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx" event={"ID":"c134796e-9f06-4068-b8a8-875673cb5570","Type":"ContainerDied","Data":"5f8321b1f49a4a2a6982b8d484b3ad59f32d251b1031dbc6cbde39e216b95a0b"}
Nov 02 14:03:46.840013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:46.839883    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f8321b1f49a4a2a6982b8d484b3ad59f32d251b1031dbc6cbde39e216b95a0b"
Nov 02 14:03:51.429059 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:03:51.429015    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:03:55.037553 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:55.037519    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"]
Nov 02 14:03:55.053848 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:55.053820    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0699e85-kdwb7"]
Nov 02 14:03:55.540915 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:55.540872    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bb0d6b-d72c-4b36-b793-3b1ec339f869" path="/var/lib/kubelet/pods/98bb0d6b-d72c-4b36-b793-3b1ec339f869/volumes"
Nov 02 14:03:59.027290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:59.027261    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"]
Nov 02 14:03:59.031618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:59.031594    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b03ef32-wfzng"]
Nov 02 14:03:59.541875 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:03:59.541843    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265c9673-ae2d-4f7d-b93e-bc35eae5b5c1" path="/var/lib/kubelet/pods/265c9673-ae2d-4f7d-b93e-bc35eae5b5c1/volumes"
Nov 02 14:04:01.032027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:01.031989    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"]
Nov 02 14:04:01.040059 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:01.040025    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a2a5c4-mlmss"]
Nov 02 14:04:01.541653 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:01.541619    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e6076b-990e-4c31-a445-7f35cf72ecdb" path="/var/lib/kubelet/pods/c6e6076b-990e-4c31-a445-7f35cf72ecdb/volumes"
Nov 02 14:04:02.028024 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:02.027983    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"]
Nov 02 14:04:02.030540 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:02.030516    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97339148-qwgb2"]
Nov 02 14:04:02.032592 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:02.032575    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"]
Nov 02 14:04:02.035853 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:02.035829    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f822813-w45nz"]
Nov 02 14:04:03.542110 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:03.542079    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a003a3f-c510-442b-8f3a-ff22263e662d" path="/var/lib/kubelet/pods/7a003a3f-c510-442b-8f3a-ff22263e662d/volumes"
Nov 02 14:04:03.542483 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:03.542455    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937a2b67-97c5-4988-a623-3bcb830c8e33" path="/var/lib/kubelet/pods/937a2b67-97c5-4988-a623-3bcb830c8e33/volumes"
Nov 02 14:04:03.998145 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:03.998112    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"]
Nov 02 14:04:03.998614 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:03.998596    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c134796e-9f06-4068-b8a8-875673cb5570" containerName="revoke-keys"
Nov 02 14:04:03.998614 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:03.998615    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c134796e-9f06-4068-b8a8-875673cb5570" containerName="revoke-keys"
Nov 02 14:04:03.998782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:03.998699    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c134796e-9f06-4068-b8a8-875673cb5570" containerName="revoke-keys"
Nov 02 14:04:04.003131 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.003115    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.009448 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.009427    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:04:04.009727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.009706    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:04:04.011512 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.011488    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"]
Nov 02 14:04:04.019534 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.019512    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15c2749-6586-4947-830f-c669654a31b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.019676 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.019551    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcwj\" (UniqueName: \"kubernetes.io/projected/b15c2749-6586-4947-830f-c669654a31b2-kube-api-access-4pcwj\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.019676 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.019660    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15c2749-6586-4947-830f-c669654a31b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.120998 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.120962    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcwj\" (UniqueName: \"kubernetes.io/projected/b15c2749-6586-4947-830f-c669654a31b2-kube-api-access-4pcwj\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.121181 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.121008    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15c2749-6586-4947-830f-c669654a31b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.121181 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.121134    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15c2749-6586-4947-830f-c669654a31b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.121714 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.121691    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15c2749-6586-4947-830f-c669654a31b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.123650 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.123627    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15c2749-6586-4947-830f-c669654a31b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.133516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.133496    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcwj\" (UniqueName: \"kubernetes.io/projected/b15c2749-6586-4947-830f-c669654a31b2-kube-api-access-4pcwj\") pod \"maas-api-revoke-sub-ae594979-bftvs\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.322189 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.322160    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:04.600767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:04.600745    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"]
Nov 02 14:04:04.602679 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:04:04.602651    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15c2749_6586_4947_830f_c669654a31b2.slice/crio-b7b3c22266e332c4faabaf3a1ba350ee2debd2e538479bc2d69e8b9818c1f807 WatchSource:0}: Error finding container b7b3c22266e332c4faabaf3a1ba350ee2debd2e538479bc2d69e8b9818c1f807: Status 404 returned error can't find the container with id b7b3c22266e332c4faabaf3a1ba350ee2debd2e538479bc2d69e8b9818c1f807
Nov 02 14:04:05.003327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:05.003288    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs" event={"ID":"b15c2749-6586-4947-830f-c669654a31b2","Type":"ContainerStarted","Data":"5a810c934d6693c2b10a38eaf43374f2f8d024edf22bc245de9c2c4bf8cbe6d1"}
Nov 02 14:04:05.003457 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:05.003377    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs" event={"ID":"b15c2749-6586-4947-830f-c669654a31b2","Type":"ContainerStarted","Data":"b7b3c22266e332c4faabaf3a1ba350ee2debd2e538479bc2d69e8b9818c1f807"}
Nov 02 14:04:05.022746 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:05.022698    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs" podStartSLOduration=2.022685627 podStartE2EDuration="2.022685627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:05.021310114 +0000 UTC m=+1221.599398981" watchObservedRunningTime="2026-11-02 14:04:05.022685627 +0000 UTC m=+1221.600774493"
Nov 02 14:04:06.011358 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:06.011322    5572 generic.go:356] "Generic (PLEG): container finished" podID="b15c2749-6586-4947-830f-c669654a31b2" containerID="5a810c934d6693c2b10a38eaf43374f2f8d024edf22bc245de9c2c4bf8cbe6d1" exitCode=0
Nov 02 14:04:06.011793 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:06.011411    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs" event={"ID":"b15c2749-6586-4947-830f-c669654a31b2","Type":"ContainerDied","Data":"5a810c934d6693c2b10a38eaf43374f2f8d024edf22bc245de9c2c4bf8cbe6d1"}
Nov 02 14:04:07.149327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.149302    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:07.251571 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.251533    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b15c2749-6586-4947-830f-c669654a31b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15c2749-6586-4947-830f-c669654a31b2-maas-api-xks-ca\") pod \"b15c2749-6586-4947-830f-c669654a31b2\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") "
Nov 02 14:04:07.251726 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.251611    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15c2749-6586-4947-830f-c669654a31b2-kube-api-access-4pcwj\" (UniqueName: \"kubernetes.io/projected/b15c2749-6586-4947-830f-c669654a31b2-kube-api-access-4pcwj\") pod \"b15c2749-6586-4947-830f-c669654a31b2\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") "
Nov 02 14:04:07.251726 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.251686    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15c2749-6586-4947-830f-c669654a31b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15c2749-6586-4947-830f-c669654a31b2-maas-api-openshift-ca\") pod \"b15c2749-6586-4947-830f-c669654a31b2\" (UID: \"b15c2749-6586-4947-830f-c669654a31b2\") "
Nov 02 14:04:07.252061 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.252036    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15c2749-6586-4947-830f-c669654a31b2-maas-api-openshift-ca" pod "b15c2749-6586-4947-830f-c669654a31b2" (UID: "b15c2749-6586-4947-830f-c669654a31b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:07.253908 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.253875    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b15c2749-6586-4947-830f-c669654a31b2-maas-api-xks-ca" pod "b15c2749-6586-4947-830f-c669654a31b2" (UID: "b15c2749-6586-4947-830f-c669654a31b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:04:07.253995 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.253974    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15c2749-6586-4947-830f-c669654a31b2-kube-api-access-4pcwj" pod "b15c2749-6586-4947-830f-c669654a31b2" (UID: "b15c2749-6586-4947-830f-c669654a31b2"). InnerVolumeSpecName "kube-api-access-4pcwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:07.353064 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.353037    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15c2749-6586-4947-830f-c669654a31b2-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:07.353064 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.353061    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15c2749-6586-4947-830f-c669654a31b2-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:07.353191 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:07.353070    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pcwj\" (UniqueName: \"kubernetes.io/projected/b15c2749-6586-4947-830f-c669654a31b2-kube-api-access-4pcwj\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:08.025636 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:08.025596    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"
Nov 02 14:04:08.025636 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:08.025612    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs" event={"ID":"b15c2749-6586-4947-830f-c669654a31b2","Type":"ContainerDied","Data":"b7b3c22266e332c4faabaf3a1ba350ee2debd2e538479bc2d69e8b9818c1f807"}
Nov 02 14:04:08.025897 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:08.025658    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7b3c22266e332c4faabaf3a1ba350ee2debd2e538479bc2d69e8b9818c1f807"
Nov 02 14:04:12.025382 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:12.025282    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"]
Nov 02 14:04:12.028136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:12.028112    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f833d2-xwhwq"]
Nov 02 14:04:13.072770 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:13.072739    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"]
Nov 02 14:04:13.075588 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:13.075569    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded92992-qt2rs"]
Nov 02 14:04:13.541521 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:13.541453    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084a1059-63a8-4a2e-b6a0-29c3c7b77bf8" path="/var/lib/kubelet/pods/084a1059-63a8-4a2e-b6a0-29c3c7b77bf8/volumes"
Nov 02 14:04:13.541881 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:13.541867    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9866232a-c748-4d81-a74f-1bc790f84bcc" path="/var/lib/kubelet/pods/9866232a-c748-4d81-a74f-1bc790f84bcc/volumes"
Nov 02 14:04:21.429367 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:04:21.429335    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:04:22.227539 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.227502    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"]
Nov 02 14:04:22.227984 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.227969    5572 cpu_manager.go:414] "containerMap: removing container" podUID="b15c2749-6586-4947-830f-c669654a31b2" containerName="revoke-keys"
Nov 02 14:04:22.228038 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.227986    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15c2749-6586-4947-830f-c669654a31b2" containerName="revoke-keys"
Nov 02 14:04:22.228074 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.228058    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15c2749-6586-4947-830f-c669654a31b2" containerName="revoke-keys"
Nov 02 14:04:22.231247 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.231232    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.242772 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.242747    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:04:22.243030 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.243008    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:04:22.243445 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.243423    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"]
Nov 02 14:04:22.297746 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.297716    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wmxl\" (UniqueName: \"kubernetes.io/projected/25d098ec-25d9-45c6-8fcd-83d43699d548-kube-api-access-5wmxl\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.297891 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.297834    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.297891 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.297871    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.398526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.398499    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.398680 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.398551    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wmxl\" (UniqueName: \"kubernetes.io/projected/25d098ec-25d9-45c6-8fcd-83d43699d548-kube-api-access-5wmxl\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.398680 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.398630    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.399176 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.399146    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.401141 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.401114    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.411738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.411717    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wmxl\" (UniqueName: \"kubernetes.io/projected/25d098ec-25d9-45c6-8fcd-83d43699d548-kube-api-access-5wmxl\") pod \"maas-api-revoke-sub-a4aef7d7-m22kg\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.545526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.545459    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:22.826087 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:22.826062    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"]
Nov 02 14:04:22.828036 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:04:22.828000    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d098ec_25d9_45c6_8fcd_83d43699d548.slice/crio-28884570dcd10d5fa2aaf225360b6244651b09b3c6caf708859a216b336240e6 WatchSource:0}: Error finding container 28884570dcd10d5fa2aaf225360b6244651b09b3c6caf708859a216b336240e6: Status 404 returned error can't find the container with id 28884570dcd10d5fa2aaf225360b6244651b09b3c6caf708859a216b336240e6
Nov 02 14:04:23.180316 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:23.180288    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg" event={"ID":"25d098ec-25d9-45c6-8fcd-83d43699d548","Type":"ContainerStarted","Data":"0910a4a9d1ab8ca8b5cf3ef2f509e2239765ba8578bf79749c1083b8761c7ff9"}
Nov 02 14:04:23.180454 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:23.180363    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg" event={"ID":"25d098ec-25d9-45c6-8fcd-83d43699d548","Type":"ContainerStarted","Data":"28884570dcd10d5fa2aaf225360b6244651b09b3c6caf708859a216b336240e6"}
Nov 02 14:04:23.200193 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:23.200147    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg" podStartSLOduration=1.200135169 podStartE2EDuration="1.200135169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:23.198871831 +0000 UTC m=+1239.776960886" watchObservedRunningTime="2026-11-02 14:04:23.200135169 +0000 UTC m=+1239.778224035"
Nov 02 14:04:24.190674 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:24.190638    5572 generic.go:356] "Generic (PLEG): container finished" podID="25d098ec-25d9-45c6-8fcd-83d43699d548" containerID="0910a4a9d1ab8ca8b5cf3ef2f509e2239765ba8578bf79749c1083b8761c7ff9" exitCode=0
Nov 02 14:04:24.191040 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:24.190732    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg" event={"ID":"25d098ec-25d9-45c6-8fcd-83d43699d548","Type":"ContainerDied","Data":"0910a4a9d1ab8ca8b5cf3ef2f509e2239765ba8578bf79749c1083b8761c7ff9"}
Nov 02 14:04:25.024622 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.024590    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"]
Nov 02 14:04:25.027956 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.027934    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0349bcbd-h8thl"]
Nov 02 14:04:25.328019 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.327996    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:25.425120 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.425098    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d098ec-25d9-45c6-8fcd-83d43699d548-kube-api-access-5wmxl\" (UniqueName: \"kubernetes.io/projected/25d098ec-25d9-45c6-8fcd-83d43699d548-kube-api-access-5wmxl\") pod \"25d098ec-25d9-45c6-8fcd-83d43699d548\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") "
Nov 02 14:04:25.425289 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.425133    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-openshift-ca\") pod \"25d098ec-25d9-45c6-8fcd-83d43699d548\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") "
Nov 02 14:04:25.425289 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.425214    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-xks-ca\") pod \"25d098ec-25d9-45c6-8fcd-83d43699d548\" (UID: \"25d098ec-25d9-45c6-8fcd-83d43699d548\") "
Nov 02 14:04:25.425581 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.425539    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-openshift-ca" pod "25d098ec-25d9-45c6-8fcd-83d43699d548" (UID: "25d098ec-25d9-45c6-8fcd-83d43699d548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:25.427440 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.427410    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-xks-ca" pod "25d098ec-25d9-45c6-8fcd-83d43699d548" (UID: "25d098ec-25d9-45c6-8fcd-83d43699d548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:04:25.427440 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.427415    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d098ec-25d9-45c6-8fcd-83d43699d548-kube-api-access-5wmxl" pod "25d098ec-25d9-45c6-8fcd-83d43699d548" (UID: "25d098ec-25d9-45c6-8fcd-83d43699d548"). InnerVolumeSpecName "kube-api-access-5wmxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:25.526348 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.526316    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:25.526348 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.526346    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wmxl\" (UniqueName: \"kubernetes.io/projected/25d098ec-25d9-45c6-8fcd-83d43699d548-kube-api-access-5wmxl\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:25.526517 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.526357    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d098ec-25d9-45c6-8fcd-83d43699d548-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:25.540864 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:25.540837    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c06f530-d33a-4feb-a704-9ac3c38bf2c7" path="/var/lib/kubelet/pods/7c06f530-d33a-4feb-a704-9ac3c38bf2c7/volumes"
Nov 02 14:04:26.029738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:26.029705    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"]
Nov 02 14:04:26.035196 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:26.035163    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"]
Nov 02 14:04:26.038045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:26.038025    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8eb94677-sd7g5"]
Nov 02 14:04:26.040159 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:26.040141    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372cd3d-5sm4d"]
Nov 02 14:04:26.204483 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:26.204455    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"
Nov 02 14:04:26.204683 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:26.204483    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg" event={"ID":"25d098ec-25d9-45c6-8fcd-83d43699d548","Type":"ContainerDied","Data":"28884570dcd10d5fa2aaf225360b6244651b09b3c6caf708859a216b336240e6"}
Nov 02 14:04:26.204683 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:26.204543    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28884570dcd10d5fa2aaf225360b6244651b09b3c6caf708859a216b336240e6"
Nov 02 14:04:27.541761 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:27.541731    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32943f10-f078-4c19-b12f-1eca8c092ff2" path="/var/lib/kubelet/pods/32943f10-f078-4c19-b12f-1eca8c092ff2/volumes"
Nov 02 14:04:27.542143 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:27.542103    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d873a3f-39ba-4af5-9076-4b34712d8a39" path="/var/lib/kubelet/pods/8d873a3f-39ba-4af5-9076-4b34712d8a39/volumes"
Nov 02 14:04:36.027706 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:36.027672    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"]
Nov 02 14:04:36.029746 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:36.029719    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06eba304-6s45v"]
Nov 02 14:04:37.554334 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:37.554300    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdbc41f-73f1-4093-8340-d0b621786ae6" path="/var/lib/kubelet/pods/bbdbc41f-73f1-4093-8340-d0b621786ae6/volumes"
Nov 02 14:04:40.986327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:40.986297    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"]
Nov 02 14:04:40.986806 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:40.986791    5572 cpu_manager.go:414] "containerMap: removing container" podUID="25d098ec-25d9-45c6-8fcd-83d43699d548" containerName="revoke-keys"
Nov 02 14:04:40.986848 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:40.986807    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d098ec-25d9-45c6-8fcd-83d43699d548" containerName="revoke-keys"
Nov 02 14:04:40.986884 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:40.986869    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d098ec-25d9-45c6-8fcd-83d43699d548" containerName="revoke-keys"
Nov 02 14:04:40.990179 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:40.990162    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:40.997838 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:40.997820    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:04:40.997934 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:40.997834    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:04:41.000025 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.000003    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"]
Nov 02 14:04:41.087728 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.087701    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.087848 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.087742    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.087848 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.087768    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xn6c\" (UniqueName: \"kubernetes.io/projected/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-kube-api-access-6xn6c\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.189431 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.189390    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.189662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.189444    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.189662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.189488    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xn6c\" (UniqueName: \"kubernetes.io/projected/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-kube-api-access-6xn6c\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.190049 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.190022    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.191962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.191937    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.201951 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.201932    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xn6c\" (UniqueName: \"kubernetes.io/projected/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-kube-api-access-6xn6c\") pod \"maas-api-revoke-sub-79805ca9-j7hfp\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.305176 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.305116    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:41.580369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:41.580346    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"]
Nov 02 14:04:41.582076 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:04:41.582049    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab50c40_d66d_4caf_94e4_aba7f1d6ddcf.slice/crio-76c169780171eb7edc287e20ce3e99b6affbfaed42258b58c57714926aa96267 WatchSource:0}: Error finding container 76c169780171eb7edc287e20ce3e99b6affbfaed42258b58c57714926aa96267: Status 404 returned error can't find the container with id 76c169780171eb7edc287e20ce3e99b6affbfaed42258b58c57714926aa96267
Nov 02 14:04:42.029690 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:42.029656    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"]
Nov 02 14:04:42.032623 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:42.032598    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c658ee07-4txhw"]
Nov 02 14:04:42.305969 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:42.305943    5572 generic.go:356] "Generic (PLEG): container finished" podID="aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" containerID="59607cffccff882c25425224bf966210699adda0562e7c0c30df79263a374526" exitCode=0
Nov 02 14:04:42.306072 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:42.306026    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp" event={"ID":"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf","Type":"ContainerDied","Data":"59607cffccff882c25425224bf966210699adda0562e7c0c30df79263a374526"}
Nov 02 14:04:42.306120 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:42.306082    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp" event={"ID":"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf","Type":"ContainerStarted","Data":"76c169780171eb7edc287e20ce3e99b6affbfaed42258b58c57714926aa96267"}
Nov 02 14:04:43.446172 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.446150    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:43.541827 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.541791    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996429a6-9cfc-4a67-9905-02c678287bb5" path="/var/lib/kubelet/pods/996429a6-9cfc-4a67-9905-02c678287bb5/volumes"
Nov 02 14:04:43.612917 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.612848    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-openshift-ca\") pod \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") "
Nov 02 14:04:43.612917 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.612895    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-xks-ca\") pod \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") "
Nov 02 14:04:43.613107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.612926    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-kube-api-access-6xn6c\" (UniqueName: \"kubernetes.io/projected/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-kube-api-access-6xn6c\") pod \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\" (UID: \"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf\") "
Nov 02 14:04:43.613197 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.613167    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-openshift-ca" pod "aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" (UID: "aab50c40-d66d-4caf-94e4-aba7f1d6ddcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:43.615667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.615640    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-xks-ca" pod "aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" (UID: "aab50c40-d66d-4caf-94e4-aba7f1d6ddcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:04:43.615792 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.615679    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-kube-api-access-6xn6c" pod "aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" (UID: "aab50c40-d66d-4caf-94e4-aba7f1d6ddcf"). InnerVolumeSpecName "kube-api-access-6xn6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:43.714335 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.714311    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xn6c\" (UniqueName: \"kubernetes.io/projected/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-kube-api-access-6xn6c\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:43.714335 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.714333    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:43.714488 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:43.714343    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:04:44.319888 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.319862    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"
Nov 02 14:04:44.320064 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.319904    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp" event={"ID":"aab50c40-d66d-4caf-94e4-aba7f1d6ddcf","Type":"ContainerDied","Data":"76c169780171eb7edc287e20ce3e99b6affbfaed42258b58c57714926aa96267"}
Nov 02 14:04:44.320064 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.319954    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c169780171eb7edc287e20ce3e99b6affbfaed42258b58c57714926aa96267"
Nov 02 14:04:44.450531 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.450497    5572 scope.go:122] "RemoveContainer" containerID="5bacd5938f87e9c43131f58f0b2ee99b665699019b66e6a7ccc2bd8a39e6c231"
Nov 02 14:04:44.459767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.459749    5572 scope.go:122] "RemoveContainer" containerID="bb810230887e8ddfd86a55436f2eaf0ad7ae9be91de6f82cf42685fb3febf892"
Nov 02 14:04:44.468407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.468387    5572 scope.go:122] "RemoveContainer" containerID="85a36edcca6848f05a15accdfec1742903cd6dd92873681976004015549f899b"
Nov 02 14:04:44.476650 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.476625    5572 scope.go:122] "RemoveContainer" containerID="34085923cb928a13c1e4bf1399983674c893f2b7342e506ad422b89bff978b14"
Nov 02 14:04:44.487887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.487586    5572 scope.go:122] "RemoveContainer" containerID="0ebada1599572d3d64daa8e77c8426c69ec3405b578aa890ac070629feaff073"
Nov 02 14:04:44.496424 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:44.496406    5572 scope.go:122] "RemoveContainer" containerID="ef5ae095a02262e498ffdd19384c9ff5e07344916f00ab40c15fc9bc3386bebd"
Nov 02 14:04:48.036178 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:48.036148    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"]
Nov 02 14:04:48.038709 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:48.038679    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"]
Nov 02 14:04:48.040501 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:48.040474    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f367ae-sjt2d"]
Nov 02 14:04:48.044123 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:48.044101    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5a19c8-hhpxd"]
Nov 02 14:04:49.541408 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:49.541367    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1989f9-fef7-4786-987f-1c913e281f90" path="/var/lib/kubelet/pods/6e1989f9-fef7-4786-987f-1c913e281f90/volumes"
Nov 02 14:04:49.541977 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:49.541955    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1d9be0-1e57-478a-89a7-b7a90d53bb39" path="/var/lib/kubelet/pods/db1d9be0-1e57-478a-89a7-b7a90d53bb39/volumes"
Nov 02 14:04:50.035406 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:50.035374    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"]
Nov 02 14:04:50.040988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:50.040953    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"]
Nov 02 14:04:50.042976 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:50.042949    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-718ada8f-799c5"]
Nov 02 14:04:50.044639 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:50.044621    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19350488-nzsqt"]
Nov 02 14:04:51.428819 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:04:51.428785    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:04:51.541369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:51.541337    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bdaa6b-c6d6-482b-9666-88d140d3057d" path="/var/lib/kubelet/pods/20bdaa6b-c6d6-482b-9666-88d140d3057d/volumes"
Nov 02 14:04:51.541771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:51.541757    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c3bd16-cf04-462f-bfe7-c5b2a5a318da" path="/var/lib/kubelet/pods/d6c3bd16-cf04-462f-bfe7-c5b2a5a318da/volumes"
Nov 02 14:04:55.058364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.058285    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"]
Nov 02 14:04:55.066750 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.066726    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"]
Nov 02 14:04:55.084334 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.084316    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"]
Nov 02 14:04:55.085948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.085929    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9ba7e3-wzxrq"]
Nov 02 14:04:55.100138 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.100104    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e420e1df-xc694"]
Nov 02 14:04:55.101728 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.101706    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fcb3f-4wd9l"]
Nov 02 14:04:55.541420 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.541330    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197ce710-8c1b-4e8e-9c1a-3e6810517f23" path="/var/lib/kubelet/pods/197ce710-8c1b-4e8e-9c1a-3e6810517f23/volumes"
Nov 02 14:04:55.541843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.541822    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9972e7d7-30bf-43f5-8b3d-4783d268c9f9" path="/var/lib/kubelet/pods/9972e7d7-30bf-43f5-8b3d-4783d268c9f9/volumes"
Nov 02 14:04:55.542259 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:04:55.542240    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a2af9c-a923-48a4-af56-a0b66084ead1" path="/var/lib/kubelet/pods/a0a2af9c-a923-48a4-af56-a0b66084ead1/volumes"
Nov 02 14:05:07.757022 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.756988    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"]
Nov 02 14:05:07.758286 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.758260    5572 cpu_manager.go:414] "containerMap: removing container" podUID="aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" containerName="revoke-keys"
Nov 02 14:05:07.758452 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.758437    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" containerName="revoke-keys"
Nov 02 14:05:07.758706 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.758689    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" containerName="revoke-keys"
Nov 02 14:05:07.762449 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.762429    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.772761 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.772734    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"]
Nov 02 14:05:07.773999 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.773978    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:05:07.774329 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.774309    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:05:07.857199 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.857158    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.857419 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.857210    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.857419 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.857293    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85p65\" (UniqueName: \"kubernetes.io/projected/a8e51270-09e2-44ef-9913-8823964b9bf4-kube-api-access-85p65\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.958024 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.957991    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.958024 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.958026    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.958234 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.958045    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85p65\" (UniqueName: \"kubernetes.io/projected/a8e51270-09e2-44ef-9913-8823964b9bf4-kube-api-access-85p65\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.958624 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.958605    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.960385 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.960364    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:07.970753 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:07.970733    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85p65\" (UniqueName: \"kubernetes.io/projected/a8e51270-09e2-44ef-9913-8823964b9bf4-kube-api-access-85p65\") pod \"maas-api-revoke-sub-0b7e8edb-wnbnj\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:08.112695 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:08.112660    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:08.397834 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:08.397801    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"]
Nov 02 14:05:08.399177 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:05:08.399152    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e51270_09e2_44ef_9913_8823964b9bf4.slice/crio-9225bfb3cabed5262b34c339a09715cb8961820c4c88dc1b74463af8e423c1e8 WatchSource:0}: Error finding container 9225bfb3cabed5262b34c339a09715cb8961820c4c88dc1b74463af8e423c1e8: Status 404 returned error can't find the container with id 9225bfb3cabed5262b34c339a09715cb8961820c4c88dc1b74463af8e423c1e8
Nov 02 14:05:08.475603 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:08.475548    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj" event={"ID":"a8e51270-09e2-44ef-9913-8823964b9bf4","Type":"ContainerStarted","Data":"9225bfb3cabed5262b34c339a09715cb8961820c4c88dc1b74463af8e423c1e8"}
Nov 02 14:05:09.483043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:09.482974    5572 generic.go:356] "Generic (PLEG): container finished" podID="a8e51270-09e2-44ef-9913-8823964b9bf4" containerID="11f6501b11877bb111068be4f677a912cde969546fff58dba281120fe0c325c5" exitCode=0
Nov 02 14:05:09.483379 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:09.483053    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj" event={"ID":"a8e51270-09e2-44ef-9913-8823964b9bf4","Type":"ContainerDied","Data":"11f6501b11877bb111068be4f677a912cde969546fff58dba281120fe0c325c5"}
Nov 02 14:05:10.616771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.616747    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:10.685257 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.685229    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-xks-ca\") pod \"a8e51270-09e2-44ef-9913-8823964b9bf4\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") "
Nov 02 14:05:10.685376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.685271    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e51270-09e2-44ef-9913-8823964b9bf4-kube-api-access-85p65\" (UniqueName: \"kubernetes.io/projected/a8e51270-09e2-44ef-9913-8823964b9bf4-kube-api-access-85p65\") pod \"a8e51270-09e2-44ef-9913-8823964b9bf4\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") "
Nov 02 14:05:10.685490 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.685470    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-openshift-ca\") pod \"a8e51270-09e2-44ef-9913-8823964b9bf4\" (UID: \"a8e51270-09e2-44ef-9913-8823964b9bf4\") "
Nov 02 14:05:10.685814 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.685791    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-openshift-ca" pod "a8e51270-09e2-44ef-9913-8823964b9bf4" (UID: "a8e51270-09e2-44ef-9913-8823964b9bf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:05:10.687430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.687402    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-xks-ca" pod "a8e51270-09e2-44ef-9913-8823964b9bf4" (UID: "a8e51270-09e2-44ef-9913-8823964b9bf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:05:10.687552 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.687531    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e51270-09e2-44ef-9913-8823964b9bf4-kube-api-access-85p65" pod "a8e51270-09e2-44ef-9913-8823964b9bf4" (UID: "a8e51270-09e2-44ef-9913-8823964b9bf4"). InnerVolumeSpecName "kube-api-access-85p65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:05:10.786962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.786889    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:10.786962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.786918    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e51270-09e2-44ef-9913-8823964b9bf4-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:10.786962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:10.786928    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85p65\" (UniqueName: \"kubernetes.io/projected/a8e51270-09e2-44ef-9913-8823964b9bf4-kube-api-access-85p65\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:11.497207 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:11.497176    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"
Nov 02 14:05:11.497359 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:11.497178    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj" event={"ID":"a8e51270-09e2-44ef-9913-8823964b9bf4","Type":"ContainerDied","Data":"9225bfb3cabed5262b34c339a09715cb8961820c4c88dc1b74463af8e423c1e8"}
Nov 02 14:05:11.497359 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:11.497290    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9225bfb3cabed5262b34c339a09715cb8961820c4c88dc1b74463af8e423c1e8"
Nov 02 14:05:14.085805 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:14.085775    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"]
Nov 02 14:05:14.094192 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:14.094163    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b6660c-kfbl9"]
Nov 02 14:05:15.540860 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:15.540831    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f778bc74-8a4e-4a4a-9321-594b2f6c3ca5" path="/var/lib/kubelet/pods/f778bc74-8a4e-4a4a-9321-594b2f6c3ca5/volumes"
Nov 02 14:05:18.453258 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.453222    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"]
Nov 02 14:05:18.453706 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.453674    5572 cpu_manager.go:414] "containerMap: removing container" podUID="a8e51270-09e2-44ef-9913-8823964b9bf4" containerName="revoke-keys"
Nov 02 14:05:18.453706 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.453689    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e51270-09e2-44ef-9913-8823964b9bf4" containerName="revoke-keys"
Nov 02 14:05:18.453789 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.453761    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e51270-09e2-44ef-9913-8823964b9bf4" containerName="revoke-keys"
Nov 02 14:05:18.458035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.458013    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.465472 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.465447    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:05:18.465814 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.465512    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:05:18.468491 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.468464    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"]
Nov 02 14:05:18.553779 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.553747    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnsq\" (UniqueName: \"kubernetes.io/projected/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-kube-api-access-glnsq\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.553950 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.553883    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.554010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.553949    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.654873 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.654846    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnsq\" (UniqueName: \"kubernetes.io/projected/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-kube-api-access-glnsq\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.655063 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.654924    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.655063 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.654974    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.655605 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.655580    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.657251 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.657236    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.667406 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.667387    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnsq\" (UniqueName: \"kubernetes.io/projected/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-kube-api-access-glnsq\") pod \"maas-api-revoke-sub-3d4156f1-c8sbc\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:18.773792 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:18.773727    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:19.049719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:19.049694    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"]
Nov 02 14:05:19.051657 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:05:19.051624    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aeb0ec_50ec_476f_b1b2_363f31c0b0b0.slice/crio-1015186fdfa566cd3c7347a8383bae3793a92d8929725a20393314ed843dce4a WatchSource:0}: Error finding container 1015186fdfa566cd3c7347a8383bae3793a92d8929725a20393314ed843dce4a: Status 404 returned error can't find the container with id 1015186fdfa566cd3c7347a8383bae3793a92d8929725a20393314ed843dce4a
Nov 02 14:05:19.543759 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:19.543722    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc" event={"ID":"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0","Type":"ContainerStarted","Data":"2559f63961634508f21ff93830ca8c3d5817ec3070d5116e69f4dc5a5bf99a1b"}
Nov 02 14:05:19.544129 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:19.543786    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc" event={"ID":"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0","Type":"ContainerStarted","Data":"1015186fdfa566cd3c7347a8383bae3793a92d8929725a20393314ed843dce4a"}
Nov 02 14:05:19.564319 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:19.564277    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc" podStartSLOduration=1.5642636730000001 podStartE2EDuration="1.564263673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:19.562024473 +0000 UTC m=+1296.140113341" watchObservedRunningTime="2026-11-02 14:05:19.564263673 +0000 UTC m=+1296.142352579"
Nov 02 14:05:20.551621 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:20.551588    5572 generic.go:356] "Generic (PLEG): container finished" podID="95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" containerID="2559f63961634508f21ff93830ca8c3d5817ec3070d5116e69f4dc5a5bf99a1b" exitCode=0
Nov 02 14:05:20.552027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:20.551656    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc" event={"ID":"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0","Type":"ContainerDied","Data":"2559f63961634508f21ff93830ca8c3d5817ec3070d5116e69f4dc5a5bf99a1b"}
Nov 02 14:05:21.048933 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.048895    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"]
Nov 02 14:05:21.050661 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.050638    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f607f6ad-5h85m"]
Nov 02 14:05:21.429325 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:05:21.429287    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:05:21.541576 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.541542    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78d60b0-75da-4133-b1f1-b878dbd7e292" path="/var/lib/kubelet/pods/e78d60b0-75da-4133-b1f1-b878dbd7e292/volumes"
Nov 02 14:05:21.683902 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.683848    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:21.787185 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.787153    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-kube-api-access-glnsq\" (UniqueName: \"kubernetes.io/projected/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-kube-api-access-glnsq\") pod \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") "
Nov 02 14:05:21.787372 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.787237    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-openshift-ca\") pod \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") "
Nov 02 14:05:21.787372 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.787338    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-xks-ca\") pod \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\" (UID: \"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0\") "
Nov 02 14:05:21.787645 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.787621    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-openshift-ca" pod "95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" (UID: "95aeb0ec-50ec-476f-b1b2-363f31c0b0b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:05:21.789510 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.789484    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-kube-api-access-glnsq" pod "95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" (UID: "95aeb0ec-50ec-476f-b1b2-363f31c0b0b0"). InnerVolumeSpecName "kube-api-access-glnsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:05:21.789626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.789549    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-xks-ca" pod "95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" (UID: "95aeb0ec-50ec-476f-b1b2-363f31c0b0b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:05:21.888806 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.888774    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:21.888806 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.888803    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glnsq\" (UniqueName: \"kubernetes.io/projected/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-kube-api-access-glnsq\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:21.888966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:21.888815    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:22.563698 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:22.563661    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"
Nov 02 14:05:22.563698 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:22.563678    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc" event={"ID":"95aeb0ec-50ec-476f-b1b2-363f31c0b0b0","Type":"ContainerDied","Data":"1015186fdfa566cd3c7347a8383bae3793a92d8929725a20393314ed843dce4a"}
Nov 02 14:05:22.563890 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:22.563731    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1015186fdfa566cd3c7347a8383bae3793a92d8929725a20393314ed843dce4a"
Nov 02 14:05:39.700878 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.700844    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"]
Nov 02 14:05:39.704141 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.704114    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8sbc"]
Nov 02 14:05:39.735499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.735468    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"]
Nov 02 14:05:39.736136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.736116    5572 cpu_manager.go:414] "containerMap: removing container" podUID="95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" containerName="revoke-keys"
Nov 02 14:05:39.736181 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.736141    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" containerName="revoke-keys"
Nov 02 14:05:39.736264 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.736253    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" containerName="revoke-keys"
Nov 02 14:05:39.739615 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.739596    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.744902 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.744881    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:05:39.747497 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.747476    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:05:39.749719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.749693    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"]
Nov 02 14:05:39.806684 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.806653    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.806684 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.806686    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vmp\" (UniqueName: \"kubernetes.io/projected/c8e51080-2c70-4b57-ab2c-7d1aebeca952-kube-api-access-f7vmp\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.806910 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.806799    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.907579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.907521    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.907579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.907575    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vmp\" (UniqueName: \"kubernetes.io/projected/c8e51080-2c70-4b57-ab2c-7d1aebeca952-kube-api-access-f7vmp\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.907786 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.907650    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.908283 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.908265    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.909898 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.909880    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:39.920097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:39.920073    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vmp\" (UniqueName: \"kubernetes.io/projected/c8e51080-2c70-4b57-ab2c-7d1aebeca952-kube-api-access-f7vmp\") pod \"maas-api-revoke-sub-3d4156f1-gpfw6\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:40.054965 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:40.054864    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:40.334186 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:40.334163    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"]
Nov 02 14:05:40.336284 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:05:40.336257    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e51080_2c70_4b57_ab2c_7d1aebeca952.slice/crio-a07518f1fda0ce80f890a4abeeab9e3e469e7b0baa71cdfa2bed0012c8172df2 WatchSource:0}: Error finding container a07518f1fda0ce80f890a4abeeab9e3e469e7b0baa71cdfa2bed0012c8172df2: Status 404 returned error can't find the container with id a07518f1fda0ce80f890a4abeeab9e3e469e7b0baa71cdfa2bed0012c8172df2
Nov 02 14:05:40.723808 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:40.723769    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6" event={"ID":"c8e51080-2c70-4b57-ab2c-7d1aebeca952","Type":"ContainerStarted","Data":"7e21cc7c5cf91e5f06cd3b59dec89193e472c73e5f9cdc24c22d52e3e7573288"}
Nov 02 14:05:40.724314 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:40.723862    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6" event={"ID":"c8e51080-2c70-4b57-ab2c-7d1aebeca952","Type":"ContainerStarted","Data":"a07518f1fda0ce80f890a4abeeab9e3e469e7b0baa71cdfa2bed0012c8172df2"}
Nov 02 14:05:40.746715 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:40.746664    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6" podStartSLOduration=1.746651805 podStartE2EDuration="1.746651805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:40.744253401 +0000 UTC m=+1317.322342268" watchObservedRunningTime="2026-11-02 14:05:40.746651805 +0000 UTC m=+1317.324740672"
Nov 02 14:05:41.541342 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:41.541262    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95aeb0ec-50ec-476f-b1b2-363f31c0b0b0" path="/var/lib/kubelet/pods/95aeb0ec-50ec-476f-b1b2-363f31c0b0b0/volumes"
Nov 02 14:05:41.731526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:41.731490    5572 generic.go:356] "Generic (PLEG): container finished" podID="c8e51080-2c70-4b57-ab2c-7d1aebeca952" containerID="7e21cc7c5cf91e5f06cd3b59dec89193e472c73e5f9cdc24c22d52e3e7573288" exitCode=0
Nov 02 14:05:41.731933 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:41.731589    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6" event={"ID":"c8e51080-2c70-4b57-ab2c-7d1aebeca952","Type":"ContainerDied","Data":"7e21cc7c5cf91e5f06cd3b59dec89193e472c73e5f9cdc24c22d52e3e7573288"}
Nov 02 14:05:42.869681 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:42.869655    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:42.936992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:42.936964    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-xks-ca\") pod \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") "
Nov 02 14:05:42.937139 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:42.937104    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e51080-2c70-4b57-ab2c-7d1aebeca952-kube-api-access-f7vmp\" (UniqueName: \"kubernetes.io/projected/c8e51080-2c70-4b57-ab2c-7d1aebeca952-kube-api-access-f7vmp\") pod \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") "
Nov 02 14:05:42.937139 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:42.937136    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-openshift-ca\") pod \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\" (UID: \"c8e51080-2c70-4b57-ab2c-7d1aebeca952\") "
Nov 02 14:05:42.937501 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:42.937478    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-openshift-ca" pod "c8e51080-2c70-4b57-ab2c-7d1aebeca952" (UID: "c8e51080-2c70-4b57-ab2c-7d1aebeca952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:05:42.938999 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:42.938983    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-xks-ca" pod "c8e51080-2c70-4b57-ab2c-7d1aebeca952" (UID: "c8e51080-2c70-4b57-ab2c-7d1aebeca952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:05:42.939271 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:42.939257    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e51080-2c70-4b57-ab2c-7d1aebeca952-kube-api-access-f7vmp" pod "c8e51080-2c70-4b57-ab2c-7d1aebeca952" (UID: "c8e51080-2c70-4b57-ab2c-7d1aebeca952"). InnerVolumeSpecName "kube-api-access-f7vmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:05:43.038364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:43.038283    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vmp\" (UniqueName: \"kubernetes.io/projected/c8e51080-2c70-4b57-ab2c-7d1aebeca952-kube-api-access-f7vmp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:43.038364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:43.038310    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:43.038364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:43.038321    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e51080-2c70-4b57-ab2c-7d1aebeca952-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:05:43.745038 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:43.744999    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"
Nov 02 14:05:43.745242 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:43.745000    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6" event={"ID":"c8e51080-2c70-4b57-ab2c-7d1aebeca952","Type":"ContainerDied","Data":"a07518f1fda0ce80f890a4abeeab9e3e469e7b0baa71cdfa2bed0012c8172df2"}
Nov 02 14:05:43.745242 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:43.745153    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a07518f1fda0ce80f890a4abeeab9e3e469e7b0baa71cdfa2bed0012c8172df2"
Nov 02 14:05:44.541268 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.541238    5572 scope.go:122] "RemoveContainer" containerID="a77094809484a0c67dfa97fd45618ad2468f80b75be08e3245652f64aa0035d5"
Nov 02 14:05:44.550817 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.550799    5572 scope.go:122] "RemoveContainer" containerID="e3340f0d06c6f6978cd4b7a0bd6ffd2be3bcd5b4b3354d67597495cdac66e04d"
Nov 02 14:05:44.559083 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.559060    5572 scope.go:122] "RemoveContainer" containerID="5c65d96b7eabe9f7e4916d4d3d7f005eb73ced6249fac592c161ad8288c39cd4"
Nov 02 14:05:44.566942 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.566921    5572 scope.go:122] "RemoveContainer" containerID="ac651213e83067a4c53f9ecd555f0604a5e69ce8494d7beb2f7cd0ca51c42506"
Nov 02 14:05:44.574821 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.574802    5572 scope.go:122] "RemoveContainer" containerID="09eb7441d1c347dd480cee277c182c8dc66d8ab382a8dd53900a6664dfda3589"
Nov 02 14:05:44.582497 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.582482    5572 scope.go:122] "RemoveContainer" containerID="4399a989ed4c9854d4c379bf4d87768c78278d754a8bb908d3b816b204c93c9d"
Nov 02 14:05:44.591378 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.591356    5572 scope.go:122] "RemoveContainer" containerID="28d9734a93eefe4f1c76fbe6d861026337b09e44f3902f14b81c8d7e69c629e5"
Nov 02 14:05:44.599731 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.599711    5572 scope.go:122] "RemoveContainer" containerID="1b3f69be7f70e40dd6f628ce34591056b13c5fc8b3770a2243cd358aee50cce9"
Nov 02 14:05:44.607640 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.607621    5572 scope.go:122] "RemoveContainer" containerID="aad60d322ad43aabd427370bd6fd49b0a3aff3516555b327969d0ec8ddbe9440"
Nov 02 14:05:44.615959 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.615941    5572 scope.go:122] "RemoveContainer" containerID="c687e6274a6826253a694b13507bcace005d3a8a1ebf29cf40e91c2cc8a9faa8"
Nov 02 14:05:44.624170 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.624153    5572 scope.go:122] "RemoveContainer" containerID="615df1e5f07c2f7742c3f3651569d45e1b3f47bf2a8eb8fb444a9bedde0baacd"
Nov 02 14:05:44.632644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.632626    5572 scope.go:122] "RemoveContainer" containerID="7471984d5d25b468c49fd9dc57c48fd189a7129ef03a97f8434180b4b4fdd141"
Nov 02 14:05:44.642686 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.642661    5572 scope.go:122] "RemoveContainer" containerID="97a14b46252e9621b94bbd477e0f00c3df6953ecdd256a8d66252b731e408872"
Nov 02 14:05:44.651751 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.651729    5572 scope.go:122] "RemoveContainer" containerID="3c9ab4f132fc4f3615264cd8ad37954b8ebf1b0e4adb590e6cded51a20e7cd18"
Nov 02 14:05:44.661905 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.661886    5572 scope.go:122] "RemoveContainer" containerID="b542b519cafa15a4bfd75ddf14a8f0923c059de8a0fed4286fd73d1eb0eb1f36"
Nov 02 14:05:44.670510 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:44.670487    5572 scope.go:122] "RemoveContainer" containerID="7bb7c4ee08cb2d85079238d048a12ee0cb885bacdb4901b498fdaff0b78ae9f7"
Nov 02 14:05:45.033212 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:45.033178    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"]
Nov 02 14:05:45.037940 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:45.037911    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"]
Nov 02 14:05:45.045682 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:45.045657    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aefef8e6-zpd8k"]
Nov 02 14:05:45.048401 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:45.048379    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0624fc49-zx8hb"]
Nov 02 14:05:45.541942 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:45.541908    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b957e4-a089-4e30-9750-7fcf7443a881" path="/var/lib/kubelet/pods/08b957e4-a089-4e30-9750-7fcf7443a881/volumes"
Nov 02 14:05:45.542481 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:05:45.542462    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc09e29-f0f5-47c7-9419-0278ceef1f7e" path="/var/lib/kubelet/pods/bcc09e29-f0f5-47c7-9419-0278ceef1f7e/volumes"
Nov 02 14:05:51.429529 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:05:51.429492    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:06:11.881645 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.881603    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"]
Nov 02 14:06:11.882056 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.882043    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c8e51080-2c70-4b57-ab2c-7d1aebeca952" containerName="revoke-keys"
Nov 02 14:06:11.882100 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.882058    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e51080-2c70-4b57-ab2c-7d1aebeca952" containerName="revoke-keys"
Nov 02 14:06:11.882142 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.882126    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e51080-2c70-4b57-ab2c-7d1aebeca952" containerName="revoke-keys"
Nov 02 14:06:11.885266 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.885251    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:11.893347 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.893323    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:06:11.893629 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.893474    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:06:11.894454 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:11.894435    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"]
Nov 02 14:06:12.050519 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.050484    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7tfs\" (UniqueName: \"kubernetes.io/projected/325aa455-fcd7-4b76-826b-0111452f775f-kube-api-access-p7tfs\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.050727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.050541    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.050727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.050704    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.151414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.151325    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7tfs\" (UniqueName: \"kubernetes.io/projected/325aa455-fcd7-4b76-826b-0111452f775f-kube-api-access-p7tfs\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.151414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.151383    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.151650 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.151439    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.152027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.152010    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.153916 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.153894    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.164993 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.164963    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7tfs\" (UniqueName: \"kubernetes.io/projected/325aa455-fcd7-4b76-826b-0111452f775f-kube-api-access-p7tfs\") pod \"maas-api-revoke-sub-e2fc9c5b-wskcd\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.199491 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.199459    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:12.476991 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.476966    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"]
Nov 02 14:06:12.478766 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:06:12.478726    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325aa455_fcd7_4b76_826b_0111452f775f.slice/crio-f5ff8e6371d2af997563cea2765ead1fc506c7772e8e0333de82c763a1026b89 WatchSource:0}: Error finding container f5ff8e6371d2af997563cea2765ead1fc506c7772e8e0333de82c763a1026b89: Status 404 returned error can't find the container with id f5ff8e6371d2af997563cea2765ead1fc506c7772e8e0333de82c763a1026b89
Nov 02 14:06:12.913184 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.913146    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd" event={"ID":"325aa455-fcd7-4b76-826b-0111452f775f","Type":"ContainerStarted","Data":"559506a93133f9775ce47626fa1ddf40db7a1bbc524f428c4217b018ff4c7106"}
Nov 02 14:06:12.913693 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.913204    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd" event={"ID":"325aa455-fcd7-4b76-826b-0111452f775f","Type":"ContainerStarted","Data":"f5ff8e6371d2af997563cea2765ead1fc506c7772e8e0333de82c763a1026b89"}
Nov 02 14:06:12.934988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:12.934942    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd" podStartSLOduration=1.934927348 podStartE2EDuration="1.934927348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:12.931825709 +0000 UTC m=+1349.509914576" watchObservedRunningTime="2026-11-02 14:06:12.934927348 +0000 UTC m=+1349.513016214"
Nov 02 14:06:13.919667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:13.919631    5572 generic.go:356] "Generic (PLEG): container finished" podID="325aa455-fcd7-4b76-826b-0111452f775f" containerID="559506a93133f9775ce47626fa1ddf40db7a1bbc524f428c4217b018ff4c7106" exitCode=0
Nov 02 14:06:13.920055 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:13.919715    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd" event={"ID":"325aa455-fcd7-4b76-826b-0111452f775f","Type":"ContainerDied","Data":"559506a93133f9775ce47626fa1ddf40db7a1bbc524f428c4217b018ff4c7106"}
Nov 02 14:06:15.063828 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.063797    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:15.178709 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.178635    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325aa455-fcd7-4b76-826b-0111452f775f-kube-api-access-p7tfs\" (UniqueName: \"kubernetes.io/projected/325aa455-fcd7-4b76-826b-0111452f775f-kube-api-access-p7tfs\") pod \"325aa455-fcd7-4b76-826b-0111452f775f\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") "
Nov 02 14:06:15.178858 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.178748    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-openshift-ca\") pod \"325aa455-fcd7-4b76-826b-0111452f775f\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") "
Nov 02 14:06:15.178858 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.178796    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-xks-ca\") pod \"325aa455-fcd7-4b76-826b-0111452f775f\" (UID: \"325aa455-fcd7-4b76-826b-0111452f775f\") "
Nov 02 14:06:15.179152 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.179117    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-openshift-ca" pod "325aa455-fcd7-4b76-826b-0111452f775f" (UID: "325aa455-fcd7-4b76-826b-0111452f775f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:15.180853 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.180830    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-xks-ca" pod "325aa455-fcd7-4b76-826b-0111452f775f" (UID: "325aa455-fcd7-4b76-826b-0111452f775f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:06:15.180853 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.180849    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325aa455-fcd7-4b76-826b-0111452f775f-kube-api-access-p7tfs" pod "325aa455-fcd7-4b76-826b-0111452f775f" (UID: "325aa455-fcd7-4b76-826b-0111452f775f"). InnerVolumeSpecName "kube-api-access-p7tfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:15.280410 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.280376    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7tfs\" (UniqueName: \"kubernetes.io/projected/325aa455-fcd7-4b76-826b-0111452f775f-kube-api-access-p7tfs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:15.280410 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.280403    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:15.280410 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.280414    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325aa455-fcd7-4b76-826b-0111452f775f-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:15.933341 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.933310    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd" event={"ID":"325aa455-fcd7-4b76-826b-0111452f775f","Type":"ContainerDied","Data":"f5ff8e6371d2af997563cea2765ead1fc506c7772e8e0333de82c763a1026b89"}
Nov 02 14:06:15.933341 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.933329    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"
Nov 02 14:06:15.933341 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:15.933357    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5ff8e6371d2af997563cea2765ead1fc506c7772e8e0333de82c763a1026b89"
Nov 02 14:06:21.429375 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:06:21.429334    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:06:22.087846 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.087818    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"]
Nov 02 14:06:22.088235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.088223    5572 cpu_manager.go:414] "containerMap: removing container" podUID="325aa455-fcd7-4b76-826b-0111452f775f" containerName="revoke-keys"
Nov 02 14:06:22.088280 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.088239    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325aa455-fcd7-4b76-826b-0111452f775f" containerName="revoke-keys"
Nov 02 14:06:22.088356 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.088347    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="325aa455-fcd7-4b76-826b-0111452f775f" containerName="revoke-keys"
Nov 02 14:06:22.095967 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.095947    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.100800 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.100780    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:06:22.100937 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.100869    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:06:22.102335 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.102305    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"]
Nov 02 14:06:22.249300 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.249264    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.249489 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.249371    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgs25\" (UniqueName: \"kubernetes.io/projected/4478ff7c-9698-4d24-84a1-f7c96e91903d-kube-api-access-lgs25\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.249489 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.249455    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.350369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.350288    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.350369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.350343    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs25\" (UniqueName: \"kubernetes.io/projected/4478ff7c-9698-4d24-84a1-f7c96e91903d-kube-api-access-lgs25\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.350617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.350407    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.351084 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.351061    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.352858 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.352840    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.363103 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.363074    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgs25\" (UniqueName: \"kubernetes.io/projected/4478ff7c-9698-4d24-84a1-f7c96e91903d-kube-api-access-lgs25\") pod \"maas-api-revoke-sub-5a29efb3-7xwdt\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.410942 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.410918    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:22.686435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.686409    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"]
Nov 02 14:06:22.687706 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:06:22.687680    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4478ff7c_9698_4d24_84a1_f7c96e91903d.slice/crio-54f616dc59d3dcd19911519e5dae8ce6824607bcc7388101683e06281916c21c WatchSource:0}: Error finding container 54f616dc59d3dcd19911519e5dae8ce6824607bcc7388101683e06281916c21c: Status 404 returned error can't find the container with id 54f616dc59d3dcd19911519e5dae8ce6824607bcc7388101683e06281916c21c
Nov 02 14:06:22.971461 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:22.971381    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt" event={"ID":"4478ff7c-9698-4d24-84a1-f7c96e91903d","Type":"ContainerStarted","Data":"54f616dc59d3dcd19911519e5dae8ce6824607bcc7388101683e06281916c21c"}
Nov 02 14:06:23.980241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:23.980202    5572 generic.go:356] "Generic (PLEG): container finished" podID="4478ff7c-9698-4d24-84a1-f7c96e91903d" containerID="3d83709630b2ea1166c2d8519b8c175db7e231badde0cef7a11e0c82f48c6b8b" exitCode=0
Nov 02 14:06:23.980792 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:23.980341    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt" event={"ID":"4478ff7c-9698-4d24-84a1-f7c96e91903d","Type":"ContainerDied","Data":"3d83709630b2ea1166c2d8519b8c175db7e231badde0cef7a11e0c82f48c6b8b"}
Nov 02 14:06:25.359059 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.359036    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:25.375891 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.375854    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-xks-ca\") pod \"4478ff7c-9698-4d24-84a1-f7c96e91903d\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") "
Nov 02 14:06:25.376021 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.375943    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-openshift-ca\") pod \"4478ff7c-9698-4d24-84a1-f7c96e91903d\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") "
Nov 02 14:06:25.376021 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.375963    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4478ff7c-9698-4d24-84a1-f7c96e91903d-kube-api-access-lgs25\" (UniqueName: \"kubernetes.io/projected/4478ff7c-9698-4d24-84a1-f7c96e91903d-kube-api-access-lgs25\") pod \"4478ff7c-9698-4d24-84a1-f7c96e91903d\" (UID: \"4478ff7c-9698-4d24-84a1-f7c96e91903d\") "
Nov 02 14:06:25.376303 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.376278    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-openshift-ca" pod "4478ff7c-9698-4d24-84a1-f7c96e91903d" (UID: "4478ff7c-9698-4d24-84a1-f7c96e91903d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:25.378043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.378021    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-xks-ca" pod "4478ff7c-9698-4d24-84a1-f7c96e91903d" (UID: "4478ff7c-9698-4d24-84a1-f7c96e91903d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:06:25.378246 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.378223    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4478ff7c-9698-4d24-84a1-f7c96e91903d-kube-api-access-lgs25" pod "4478ff7c-9698-4d24-84a1-f7c96e91903d" (UID: "4478ff7c-9698-4d24-84a1-f7c96e91903d"). InnerVolumeSpecName "kube-api-access-lgs25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:25.476760 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.476731    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:25.476760 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.476754    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4478ff7c-9698-4d24-84a1-f7c96e91903d-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:25.476760 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:25.476765    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgs25\" (UniqueName: \"kubernetes.io/projected/4478ff7c-9698-4d24-84a1-f7c96e91903d-kube-api-access-lgs25\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:26.021983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:26.021908    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"
Nov 02 14:06:26.021983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:26.021928    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt" event={"ID":"4478ff7c-9698-4d24-84a1-f7c96e91903d","Type":"ContainerDied","Data":"54f616dc59d3dcd19911519e5dae8ce6824607bcc7388101683e06281916c21c"}
Nov 02 14:06:26.021983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:26.021975    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54f616dc59d3dcd19911519e5dae8ce6824607bcc7388101683e06281916c21c"
Nov 02 14:06:30.033009 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:30.032969    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"]
Nov 02 14:06:30.042488 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:30.042459    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fkx6b"]
Nov 02 14:06:31.542681 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:31.542646    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34482c0e-1daa-4d1f-adff-b0f3a62ba178" path="/var/lib/kubelet/pods/34482c0e-1daa-4d1f-adff-b0f3a62ba178/volumes"
Nov 02 14:06:44.753276 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.753243    5572 scope.go:122] "RemoveContainer" containerID="e701253ff101ddfb9040de432f1b675b44264a21475849b6b635acbbeee369e4"
Nov 02 14:06:44.762325 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.762302    5572 scope.go:122] "RemoveContainer" containerID="f3c5ab0b14fca07937fd965647e990d85cd8fb41d657a0174120953862c59802"
Nov 02 14:06:44.770380 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.770363    5572 scope.go:122] "RemoveContainer" containerID="73f41a850dda6b3db4e7400d17728c4d45ecc8e25e28f8fb9ac267666470241d"
Nov 02 14:06:44.778275 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.778252    5572 scope.go:122] "RemoveContainer" containerID="8f48fa744f60785b55424c165d5115554066d1af9ed9b8ff0a8c3b3283ca4c99"
Nov 02 14:06:44.786107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.786087    5572 scope.go:122] "RemoveContainer" containerID="6cf740e3cdcd555c560a78a0bfe4f06b0263aadc161581c4de67434f2b4ef66b"
Nov 02 14:06:44.793963 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.793946    5572 scope.go:122] "RemoveContainer" containerID="e636d826a8119f69b78365f6afed8468b1b9e80aab932686b395b05e43c8c831"
Nov 02 14:06:44.801426 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.801412    5572 scope.go:122] "RemoveContainer" containerID="3c713db6a82c03d2784fd1efa95a6666ce0e3fcfd6fe2e2be18af59f57e751de"
Nov 02 14:06:44.809227 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.809209    5572 scope.go:122] "RemoveContainer" containerID="5c09a63028ff1098feb9f5d7319c19445b873df598947603ca2ffba0aebc047d"
Nov 02 14:06:44.816957 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.816940    5572 scope.go:122] "RemoveContainer" containerID="517dd8b0ff53c1e00ff1eae660a65dd6ddbff0998b530bf8e967109b9ad3e287"
Nov 02 14:06:44.824865 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.824840    5572 scope.go:122] "RemoveContainer" containerID="1e4611cd92d2901ca8b1b96b94e9249702242849af32266158b4a76135c74b8d"
Nov 02 14:06:44.833145 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.833128    5572 scope.go:122] "RemoveContainer" containerID="f0cd4bcca44e197609ae37b5b5c7f62cd245c5116a7d45e184202e2d1d435715"
Nov 02 14:06:44.840607 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.840585    5572 scope.go:122] "RemoveContainer" containerID="dcb5b05d15feae78292dcdb46904cdbe15757ec9464fb03b9df1fbd1b401b28e"
Nov 02 14:06:44.847906 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:44.847890    5572 scope.go:122] "RemoveContainer" containerID="d1f50a9fae125e0e20065bc2633f50ac7173bce847556c15f23038b7c8f8cb6e"
Nov 02 14:06:49.893209 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.893170    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"]
Nov 02 14:06:49.899854 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.899829    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gpfw6"]
Nov 02 14:06:49.921837 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.921803    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"]
Nov 02 14:06:49.922257 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.922242    5572 cpu_manager.go:414] "containerMap: removing container" podUID="4478ff7c-9698-4d24-84a1-f7c96e91903d" containerName="revoke-keys"
Nov 02 14:06:49.922313 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.922258    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4478ff7c-9698-4d24-84a1-f7c96e91903d" containerName="revoke-keys"
Nov 02 14:06:49.922359 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.922350    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4478ff7c-9698-4d24-84a1-f7c96e91903d" containerName="revoke-keys"
Nov 02 14:06:49.927789 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.927769    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:49.936437 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:49.935613    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"]
Nov 02 14:06:50.001712 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.001683    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvs6\" (UniqueName: \"kubernetes.io/projected/62ab78a8-e4c6-4699-bf52-2f1de35609a8-kube-api-access-xbvs6\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.001900 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.001743    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.001900 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.001866    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.003911 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.003890    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:06:50.004094 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.004078    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:06:50.102502 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.102474    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvs6\" (UniqueName: \"kubernetes.io/projected/62ab78a8-e4c6-4699-bf52-2f1de35609a8-kube-api-access-xbvs6\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.102651 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.102535    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.102651 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.102579    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.103238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.103215    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.105022 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.104999    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.114950 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.114927    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvs6\" (UniqueName: \"kubernetes.io/projected/62ab78a8-e4c6-4699-bf52-2f1de35609a8-kube-api-access-xbvs6\") pod \"maas-api-revoke-sub-3d4156f1-tl8f9\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.242024 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.241951    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:50.516344 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:50.516321    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"]
Nov 02 14:06:50.518679 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:06:50.518647    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ab78a8_e4c6_4699_bf52_2f1de35609a8.slice/crio-0c586dc7c8abebe0f5fceb18b8133436c27a34d637d8cfcfaecb2fbe2467bf05 WatchSource:0}: Error finding container 0c586dc7c8abebe0f5fceb18b8133436c27a34d637d8cfcfaecb2fbe2467bf05: Status 404 returned error can't find the container with id 0c586dc7c8abebe0f5fceb18b8133436c27a34d637d8cfcfaecb2fbe2467bf05
Nov 02 14:06:51.159696 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:51.159661    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9" event={"ID":"62ab78a8-e4c6-4699-bf52-2f1de35609a8","Type":"ContainerStarted","Data":"fab5f898dd69de99defa48200188506249e1b38a959d6b7225019071d5798564"}
Nov 02 14:06:51.160079 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:51.159724    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9" event={"ID":"62ab78a8-e4c6-4699-bf52-2f1de35609a8","Type":"ContainerStarted","Data":"0c586dc7c8abebe0f5fceb18b8133436c27a34d637d8cfcfaecb2fbe2467bf05"}
Nov 02 14:06:51.179061 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:51.179022    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9" podStartSLOduration=2.179009954 podStartE2EDuration="2.179009954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:51.178329403 +0000 UTC m=+1387.756418270" watchObservedRunningTime="2026-11-02 14:06:51.179009954 +0000 UTC m=+1387.757098863"
Nov 02 14:06:51.429318 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:06:51.429234    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:06:51.541368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:51.541338    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e51080-2c70-4b57-ab2c-7d1aebeca952" path="/var/lib/kubelet/pods/c8e51080-2c70-4b57-ab2c-7d1aebeca952/volumes"
Nov 02 14:06:52.168940 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:52.168907    5572 generic.go:356] "Generic (PLEG): container finished" podID="62ab78a8-e4c6-4699-bf52-2f1de35609a8" containerID="fab5f898dd69de99defa48200188506249e1b38a959d6b7225019071d5798564" exitCode=0
Nov 02 14:06:52.169328 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:52.168968    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9" event={"ID":"62ab78a8-e4c6-4699-bf52-2f1de35609a8","Type":"ContainerDied","Data":"fab5f898dd69de99defa48200188506249e1b38a959d6b7225019071d5798564"}
Nov 02 14:06:53.308097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.308070    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:53.435933 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.435858    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ab78a8-e4c6-4699-bf52-2f1de35609a8-kube-api-access-xbvs6\" (UniqueName: \"kubernetes.io/projected/62ab78a8-e4c6-4699-bf52-2f1de35609a8-kube-api-access-xbvs6\") pod \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") "
Nov 02 14:06:53.435933 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.435918    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-openshift-ca\") pod \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") "
Nov 02 14:06:53.436154 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.435943    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-xks-ca\") pod \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\" (UID: \"62ab78a8-e4c6-4699-bf52-2f1de35609a8\") "
Nov 02 14:06:53.436315 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.436293    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-openshift-ca" pod "62ab78a8-e4c6-4699-bf52-2f1de35609a8" (UID: "62ab78a8-e4c6-4699-bf52-2f1de35609a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:53.438013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.437985    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ab78a8-e4c6-4699-bf52-2f1de35609a8-kube-api-access-xbvs6" pod "62ab78a8-e4c6-4699-bf52-2f1de35609a8" (UID: "62ab78a8-e4c6-4699-bf52-2f1de35609a8"). InnerVolumeSpecName "kube-api-access-xbvs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:53.438123 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.438065    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-xks-ca" pod "62ab78a8-e4c6-4699-bf52-2f1de35609a8" (UID: "62ab78a8-e4c6-4699-bf52-2f1de35609a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:06:53.537662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.537634    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:53.537662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.537661    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab78a8-e4c6-4699-bf52-2f1de35609a8-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:53.537827 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:53.537670    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbvs6\" (UniqueName: \"kubernetes.io/projected/62ab78a8-e4c6-4699-bf52-2f1de35609a8-kube-api-access-xbvs6\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:06:54.179680 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:54.179655    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"
Nov 02 14:06:54.179851 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:54.179651    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9" event={"ID":"62ab78a8-e4c6-4699-bf52-2f1de35609a8","Type":"ContainerDied","Data":"0c586dc7c8abebe0f5fceb18b8133436c27a34d637d8cfcfaecb2fbe2467bf05"}
Nov 02 14:06:54.179851 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:06:54.179776    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c586dc7c8abebe0f5fceb18b8133436c27a34d637d8cfcfaecb2fbe2467bf05"
Nov 02 14:07:11.657512 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.657483    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"]
Nov 02 14:07:11.657928 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.657910    5572 cpu_manager.go:414] "containerMap: removing container" podUID="62ab78a8-e4c6-4699-bf52-2f1de35609a8" containerName="revoke-keys"
Nov 02 14:07:11.657928 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.657922    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ab78a8-e4c6-4699-bf52-2f1de35609a8" containerName="revoke-keys"
Nov 02 14:07:11.658025 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.658013    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ab78a8-e4c6-4699-bf52-2f1de35609a8" containerName="revoke-keys"
Nov 02 14:07:11.661092 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.661076    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.669703 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.669678    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0c51da-openshift-default-dockercfg-xmkrh\""
Nov 02 14:07:11.679839 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.679815    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"]
Nov 02 14:07:11.744876 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.744832    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-envoy\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745067 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745014    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-data\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745148 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745110    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-certs\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745270 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745232    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-credential-socket\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745280    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-socket\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745314    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-istio-token\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745343    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d46816-9d54-4328-830a-d497d35d593a-istio-podinfo\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745412    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d46816-9d54-4328-830a-d497d35d593a-istiod-ca-cert\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.745626 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.745449    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxjt\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-kube-api-access-rcxjt\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.846939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.846903    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-data\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847102 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.846947    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-certs\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847102 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847007    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-credential-socket\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847102 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847039    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-socket\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847223 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847172    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-istio-token\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847223 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847216    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d46816-9d54-4328-830a-d497d35d593a-istio-podinfo\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847317 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847295    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d46816-9d54-4328-830a-d497d35d593a-istiod-ca-cert\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847372 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847322    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-data\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847372 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847358    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-certs\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847476 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847407    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-credential-socket\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847476 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847430    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxjt\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-kube-api-access-rcxjt\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847608 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847475    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-envoy\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847608 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847543    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-socket\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.847887 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.847868    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d46816-9d54-4328-830a-d497d35d593a-istiod-ca-cert\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.858523 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.858454    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d46816-9d54-4328-830a-d497d35d593a-istio-podinfo\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.861928 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.861875    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-istio-token\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.862725 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.862699    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxjt\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-kube-api-access-rcxjt\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.865069 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.865045    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-envoy\") pod \"e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") " pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:11.974614 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:11.974525    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:12.296675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:12.296649    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"]
Nov 02 14:07:12.298823 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:07:12.298796    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d46816_9d54_4328_830a_d497d35d593a.slice/crio-6ba1c6be0719046d2cf6876aa6850a1fa0561121da485891081574b6ec1f607a WatchSource:0}: Error finding container 6ba1c6be0719046d2cf6876aa6850a1fa0561121da485891081574b6ec1f607a: Status 404 returned error can't find the container with id 6ba1c6be0719046d2cf6876aa6850a1fa0561121da485891081574b6ec1f607a
Nov 02 14:07:12.300787 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:12.300758    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:07:12.300868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:12.300824    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:07:12.300868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:12.300852    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:07:13.295120 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:13.295084    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" event={"ID":"19d46816-9d54-4328-830a-d497d35d593a","Type":"ContainerStarted","Data":"41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514"}
Nov 02 14:07:13.295651 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:13.295146    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" event={"ID":"19d46816-9d54-4328-830a-d497d35d593a","Type":"ContainerStarted","Data":"6ba1c6be0719046d2cf6876aa6850a1fa0561121da485891081574b6ec1f607a"}
Nov 02 14:07:13.322994 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:13.322941    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podStartSLOduration=2.322928567 podStartE2EDuration="2.322928567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:13.321130311 +0000 UTC m=+1409.899219179" watchObservedRunningTime="2026-11-02 14:07:13.322928567 +0000 UTC m=+1409.901017433"
Nov 02 14:07:13.975715 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:13.975682    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:13.980513 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:13.980492    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:14.300062 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:14.299985    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:14.300874 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:14.300855    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:17.213863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.213834    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"]
Nov 02 14:07:17.217636 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.217619    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.233435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.233396    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0c51da\""
Nov 02 14:07:17.235075 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.235050    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"]
Nov 02 14:07:17.301428 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.301399    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dec9d7-3765-4130-ba40-fc078a2d9b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.301549 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.301463    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj694\" (UniqueName: \"kubernetes.io/projected/66dec9d7-3765-4130-ba40-fc078a2d9b6e-kube-api-access-rj694\") pod \"payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.315080 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.315056    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"]
Nov 02 14:07:17.318612 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.318596    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.324496 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.324474    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d180e515-dockercfg-x5m6f\""
Nov 02 14:07:17.352657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.352625    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"]
Nov 02 14:07:17.402473 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.402427    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj694\" (UniqueName: \"kubernetes.io/projected/66dec9d7-3765-4130-ba40-fc078a2d9b6e-kube-api-access-rj694\") pod \"payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.402727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.402532    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.402727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.402586    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfp46\" (UniqueName: \"kubernetes.io/projected/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-kube-api-access-cfp46\") pod \"payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.402727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.402606    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dec9d7-3765-4130-ba40-fc078a2d9b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.403248 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.403224    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dec9d7-3765-4130-ba40-fc078a2d9b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.416824 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.416800    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj694\" (UniqueName: \"kubernetes.io/projected/66dec9d7-3765-4130-ba40-fc078a2d9b6e-kube-api-access-rj694\") pod \"payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.503534 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.503454    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.503534 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.503495    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfp46\" (UniqueName: \"kubernetes.io/projected/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-kube-api-access-cfp46\") pod \"payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.504144 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.504123    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.517492 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.517464    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfp46\" (UniqueName: \"kubernetes.io/projected/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-kube-api-access-cfp46\") pod \"payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.531354 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.531331    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:17.652020 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.651865    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:17.815662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.815635    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"]
Nov 02 14:07:17.818060 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:07:17.818038    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dec9d7_3765_4130_ba40_fc078a2d9b6e.slice/crio-bc8e025f93620267479b48dac9c8a3ed6fea114331337444b4dbc81db15b6f36 WatchSource:0}: Error finding container bc8e025f93620267479b48dac9c8a3ed6fea114331337444b4dbc81db15b6f36: Status 404 returned error can't find the container with id bc8e025f93620267479b48dac9c8a3ed6fea114331337444b4dbc81db15b6f36
Nov 02 14:07:17.941620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:17.941587    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"]
Nov 02 14:07:17.943412 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:07:17.943385    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b08f34_63d7_4f9c_a7ba_c81bad2094cb.slice/crio-406a63205e34692702fa4cfd8beaff83b0c53eb738f59aa9fb57f558a67ed00a WatchSource:0}: Error finding container 406a63205e34692702fa4cfd8beaff83b0c53eb738f59aa9fb57f558a67ed00a: Status 404 returned error can't find the container with id 406a63205e34692702fa4cfd8beaff83b0c53eb738f59aa9fb57f558a67ed00a
Nov 02 14:07:18.320532 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.320497    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" event={"ID":"66dec9d7-3765-4130-ba40-fc078a2d9b6e","Type":"ContainerStarted","Data":"b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219"}
Nov 02 14:07:18.321038 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.320701    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:18.321038 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.320746    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" event={"ID":"66dec9d7-3765-4130-ba40-fc078a2d9b6e","Type":"ContainerStarted","Data":"bc8e025f93620267479b48dac9c8a3ed6fea114331337444b4dbc81db15b6f36"}
Nov 02 14:07:18.322045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.322024    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" event={"ID":"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb","Type":"ContainerStarted","Data":"a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33"}
Nov 02 14:07:18.322142 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.322060    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" event={"ID":"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb","Type":"ContainerStarted","Data":"406a63205e34692702fa4cfd8beaff83b0c53eb738f59aa9fb57f558a67ed00a"}
Nov 02 14:07:18.322182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.322160    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:18.372316 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.372269    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" podStartSLOduration=1.3722565150000001 podStartE2EDuration="1.372256515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:18.369865371 +0000 UTC m=+1414.947954238" watchObservedRunningTime="2026-11-02 14:07:18.372256515 +0000 UTC m=+1414.950345382"
Nov 02 14:07:18.415008 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:18.414942    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" podStartSLOduration=1.414924715 podStartE2EDuration="1.414924715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:18.411372896 +0000 UTC m=+1414.989461780" watchObservedRunningTime="2026-11-02 14:07:18.414924715 +0000 UTC m=+1414.993013587"
Nov 02 14:07:21.433162 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:07:21.433124    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:07:29.330877 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:29.330851    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:07:29.331308 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:29.330918    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:07:29.832976 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:29.832377    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": read tcp 10.134.0.2:33726->10.134.0.160:15021: read: connection reset by peer" start-of-body=
Nov 02 14:07:29.832976 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:29.832442    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": read tcp 10.134.0.2:33726->10.134.0.160:15021: read: connection reset by peer"
Nov 02 14:07:30.399921 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:30.399885    5572 generic.go:356] "Generic (PLEG): container finished" podID="19d46816-9d54-4328-830a-d497d35d593a" containerID="41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514" exitCode=0
Nov 02 14:07:30.400333 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:30.399956    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" event={"ID":"19d46816-9d54-4328-830a-d497d35d593a","Type":"ContainerDied","Data":"41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514"}
Nov 02 14:07:30.400333 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:30.400318    5572 scope.go:122] "RemoveContainer" containerID="41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514"
Nov 02 14:07:30.401476 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:30.401451    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:07:30.401589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:30.401523    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:07:30.401589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:30.401575    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:07:31.427591 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:31.419758    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" event={"ID":"19d46816-9d54-4328-830a-d497d35d593a","Type":"ContainerStarted","Data":"cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca"}
Nov 02 14:07:31.975061 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:31.975027    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:31.975387 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:31.975363    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 14:07:31.975441 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:31.975410    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 14:07:32.975388 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:32.975351    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 14:07:32.975811 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:32.975405    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 14:07:33.975745 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:33.975705    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 14:07:33.976223 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:33.975769    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 14:07:34.975218 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:34.975183    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 14:07:34.975408 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:34.975239    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 14:07:35.975253 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:35.975210    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 14:07:35.975797 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:35.975272    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 14:07:36.978373 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:36.978338    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:36.978804 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:36.978594    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:36.979364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:36.979348    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:07:38.080001 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:38.079970    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"]
Nov 02 14:07:38.091200 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:38.091164    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d28vk"]
Nov 02 14:07:39.541918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:39.541881    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497e573e-ce57-4466-bf56-fb247d3c4efd" path="/var/lib/kubelet/pods/497e573e-ce57-4466-bf56-fb247d3c4efd/volumes"
Nov 02 14:07:44.912499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:44.912467    5572 scope.go:122] "RemoveContainer" containerID="b19cd5e1805b1e8bae198a73aee462bdd4effc236f8f88958ba50878d4c6fcf1"
Nov 02 14:07:51.429804 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:07:51.429764    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:07:59.043387 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:59.043313    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"]
Nov 02 14:07:59.048230 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:59.048202    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-pt46q"]
Nov 02 14:07:59.541647 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:07:59.541617    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efe2a3b-d746-4290-a243-d305bb3ac01c" path="/var/lib/kubelet/pods/4efe2a3b-d746-4290-a243-d305bb3ac01c/volumes"
Nov 02 14:08:02.573802 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.573766    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"]
Nov 02 14:08:02.577372 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.577355    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.585725 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.585681    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:08:02.585918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.585866    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:08:02.588678 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.588645    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"]
Nov 02 14:08:02.733317 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.733285    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.733480 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.733408    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.733480 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.733464    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6lk\" (UniqueName: \"kubernetes.io/projected/d3e8a277-5ab3-4b15-b522-e88204253424-kube-api-access-zx6lk\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.834483 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.834397    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.834483 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.834442    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6lk\" (UniqueName: \"kubernetes.io/projected/d3e8a277-5ab3-4b15-b522-e88204253424-kube-api-access-zx6lk\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.834721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.834533    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.835238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.835164    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.836839 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.836818    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.848831 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.848811    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6lk\" (UniqueName: \"kubernetes.io/projected/d3e8a277-5ab3-4b15-b522-e88204253424-kube-api-access-zx6lk\") pod \"maas-api-revoke-sub-674c462a-qk69l\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:02.892512 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:02.892490    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:03.170992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:03.170904    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"]
Nov 02 14:08:03.176362 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:08:03.176333    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e8a277_5ab3_4b15_b522_e88204253424.slice/crio-72664378d08d463d18325b5c5ac37d2fc23f9287911aca4acedd60b21041884a WatchSource:0}: Error finding container 72664378d08d463d18325b5c5ac37d2fc23f9287911aca4acedd60b21041884a: Status 404 returned error can't find the container with id 72664378d08d463d18325b5c5ac37d2fc23f9287911aca4acedd60b21041884a
Nov 02 14:08:03.606628 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:03.606591    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l" event={"ID":"d3e8a277-5ab3-4b15-b522-e88204253424","Type":"ContainerStarted","Data":"da26de0abe4d3296a379063b49262a5b5f1008c073f41f4470b19c9e59d46116"}
Nov 02 14:08:03.607216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:03.606668    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l" event={"ID":"d3e8a277-5ab3-4b15-b522-e88204253424","Type":"ContainerStarted","Data":"72664378d08d463d18325b5c5ac37d2fc23f9287911aca4acedd60b21041884a"}
Nov 02 14:08:03.635671 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:03.635629    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l" podStartSLOduration=1.635615648 podStartE2EDuration="1.635615648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:03.631862321 +0000 UTC m=+1460.209951188" watchObservedRunningTime="2026-11-02 14:08:03.635615648 +0000 UTC m=+1460.213704515"
Nov 02 14:08:04.614593 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:04.614539    5572 generic.go:356] "Generic (PLEG): container finished" podID="d3e8a277-5ab3-4b15-b522-e88204253424" containerID="da26de0abe4d3296a379063b49262a5b5f1008c073f41f4470b19c9e59d46116" exitCode=0
Nov 02 14:08:04.614986 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:04.614592    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l" event={"ID":"d3e8a277-5ab3-4b15-b522-e88204253424","Type":"ContainerDied","Data":"da26de0abe4d3296a379063b49262a5b5f1008c073f41f4470b19c9e59d46116"}
Nov 02 14:08:05.754703 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.754680    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:05.862311 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.862276    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-openshift-ca\") pod \"d3e8a277-5ab3-4b15-b522-e88204253424\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") "
Nov 02 14:08:05.862705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.862445    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e8a277-5ab3-4b15-b522-e88204253424-kube-api-access-zx6lk\" (UniqueName: \"kubernetes.io/projected/d3e8a277-5ab3-4b15-b522-e88204253424-kube-api-access-zx6lk\") pod \"d3e8a277-5ab3-4b15-b522-e88204253424\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") "
Nov 02 14:08:05.862705 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.862525    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-xks-ca\") pod \"d3e8a277-5ab3-4b15-b522-e88204253424\" (UID: \"d3e8a277-5ab3-4b15-b522-e88204253424\") "
Nov 02 14:08:05.862953 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.862930    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-openshift-ca" pod "d3e8a277-5ab3-4b15-b522-e88204253424" (UID: "d3e8a277-5ab3-4b15-b522-e88204253424"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:05.864789 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.864763    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-xks-ca" pod "d3e8a277-5ab3-4b15-b522-e88204253424" (UID: "d3e8a277-5ab3-4b15-b522-e88204253424"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:05.864881 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.864821    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e8a277-5ab3-4b15-b522-e88204253424-kube-api-access-zx6lk" pod "d3e8a277-5ab3-4b15-b522-e88204253424" (UID: "d3e8a277-5ab3-4b15-b522-e88204253424"). InnerVolumeSpecName "kube-api-access-zx6lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:05.963740 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.963710    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:05.963863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.963735    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8a277-5ab3-4b15-b522-e88204253424-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:05.963863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:05.963757    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6lk\" (UniqueName: \"kubernetes.io/projected/d3e8a277-5ab3-4b15-b522-e88204253424-kube-api-access-zx6lk\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:06.626552 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:06.626516    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l" event={"ID":"d3e8a277-5ab3-4b15-b522-e88204253424","Type":"ContainerDied","Data":"72664378d08d463d18325b5c5ac37d2fc23f9287911aca4acedd60b21041884a"}
Nov 02 14:08:06.626552 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:06.626551    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"
Nov 02 14:08:06.626810 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:06.626598    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72664378d08d463d18325b5c5ac37d2fc23f9287911aca4acedd60b21041884a"
Nov 02 14:08:10.030682 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:10.030615    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"]
Nov 02 14:08:10.033179 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:10.033141    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-s97kd"]
Nov 02 14:08:11.541140 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:11.541105    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0d94a4-78f3-4395-8b49-cc810e95a407" path="/var/lib/kubelet/pods/fe0d94a4-78f3-4395-8b49-cc810e95a407/volumes"
Nov 02 14:08:13.714935 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.714902    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"]
Nov 02 14:08:13.715377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.715363    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d3e8a277-5ab3-4b15-b522-e88204253424" containerName="revoke-keys"
Nov 02 14:08:13.715423 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.715378    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e8a277-5ab3-4b15-b522-e88204253424" containerName="revoke-keys"
Nov 02 14:08:13.715489 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.715477    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e8a277-5ab3-4b15-b522-e88204253424" containerName="revoke-keys"
Nov 02 14:08:13.718808 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.718794    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.733610 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.733590    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:08:13.733853 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.733835    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:08:13.741327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.741288    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"]
Nov 02 14:08:13.860767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.860731    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.861003 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.860868    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtsd\" (UniqueName: \"kubernetes.io/projected/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-kube-api-access-vjtsd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.861003 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.860918    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.961839 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.961802    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtsd\" (UniqueName: \"kubernetes.io/projected/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-kube-api-access-vjtsd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.962021 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.961858    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.962021 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.961948    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.962495 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.962470    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.964533 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.964511    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:13.982601 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:13.982515    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtsd\" (UniqueName: \"kubernetes.io/projected/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-kube-api-access-vjtsd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:14.031362 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:14.031343    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:14.317153 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:14.317129    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"]
Nov 02 14:08:14.671912 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:14.671878    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9" event={"ID":"fec289d0-283d-4da2-becc-8dbb7fd5e5ea","Type":"ContainerStarted","Data":"bd452a7f0ce9afe88368727f470784844d1e9ea3b8ad9e146a0bf20c19b6d6c4"}
Nov 02 14:08:14.672073 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:14.671953    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9" event={"ID":"fec289d0-283d-4da2-becc-8dbb7fd5e5ea","Type":"ContainerStarted","Data":"7bd566e162673e9a2852947af2dc835339423f1ca045a191ea3ccb2239e38b1b"}
Nov 02 14:08:14.693437 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:14.693388    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9" podStartSLOduration=1.693372374 podStartE2EDuration="1.693372374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:14.691438268 +0000 UTC m=+1471.269527136" watchObservedRunningTime="2026-11-02 14:08:14.693372374 +0000 UTC m=+1471.271461233"
Nov 02 14:08:15.683714 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:15.683671    5572 generic.go:356] "Generic (PLEG): container finished" podID="fec289d0-283d-4da2-becc-8dbb7fd5e5ea" containerID="bd452a7f0ce9afe88368727f470784844d1e9ea3b8ad9e146a0bf20c19b6d6c4" exitCode=0
Nov 02 14:08:15.684182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:15.683786    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9" event={"ID":"fec289d0-283d-4da2-becc-8dbb7fd5e5ea","Type":"ContainerDied","Data":"bd452a7f0ce9afe88368727f470784844d1e9ea3b8ad9e146a0bf20c19b6d6c4"}
Nov 02 14:08:16.824689 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.824663    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:16.887414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.887387    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-openshift-ca\") pod \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") "
Nov 02 14:08:16.887586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.887455    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-xks-ca\") pod \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") "
Nov 02 14:08:16.887644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.887612    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-kube-api-access-vjtsd\" (UniqueName: \"kubernetes.io/projected/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-kube-api-access-vjtsd\") pod \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\" (UID: \"fec289d0-283d-4da2-becc-8dbb7fd5e5ea\") "
Nov 02 14:08:16.887801 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.887781    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-openshift-ca" pod "fec289d0-283d-4da2-becc-8dbb7fd5e5ea" (UID: "fec289d0-283d-4da2-becc-8dbb7fd5e5ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:16.887993 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.887979    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:16.889526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.889505    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-xks-ca" pod "fec289d0-283d-4da2-becc-8dbb7fd5e5ea" (UID: "fec289d0-283d-4da2-becc-8dbb7fd5e5ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:16.889871 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.889852    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-kube-api-access-vjtsd" pod "fec289d0-283d-4da2-becc-8dbb7fd5e5ea" (UID: "fec289d0-283d-4da2-becc-8dbb7fd5e5ea"). InnerVolumeSpecName "kube-api-access-vjtsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:16.988751 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.988692    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjtsd\" (UniqueName: \"kubernetes.io/projected/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-kube-api-access-vjtsd\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:16.988751 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:16.988715    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec289d0-283d-4da2-becc-8dbb7fd5e5ea-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:17.695685 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:17.695650    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9" event={"ID":"fec289d0-283d-4da2-becc-8dbb7fd5e5ea","Type":"ContainerDied","Data":"7bd566e162673e9a2852947af2dc835339423f1ca045a191ea3ccb2239e38b1b"}
Nov 02 14:08:17.695685 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:17.695686    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"
Nov 02 14:08:17.695879 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:17.695707    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bd566e162673e9a2852947af2dc835339423f1ca045a191ea3ccb2239e38b1b"
Nov 02 14:08:21.429768 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:08:21.429732    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:08:23.898843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:23.898815    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"]
Nov 02 14:08:23.915669 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:23.915636    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"]
Nov 02 14:08:24.975032 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:24.975004    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"]
Nov 02 14:08:24.989312 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:24.989286    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0c51da-5scl9"]
Nov 02 14:08:25.541280 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:25.541248    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec289d0-283d-4da2-becc-8dbb7fd5e5ea" path="/var/lib/kubelet/pods/fec289d0-283d-4da2-becc-8dbb7fd5e5ea/volumes"
Nov 02 14:08:28.899953 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:28.899907    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" podUID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" containerName="payload-processing" containerID="cri-o://a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33" gracePeriod=25
Nov 02 14:08:28.915917 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:28.915878    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" podUID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" containerName="payload-pre-processing" containerID="cri-o://b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219" gracePeriod=25
Nov 02 14:08:29.329698 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:29.329657    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" podUID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:08:29.329874 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:29.329658    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" podUID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:08:32.343122 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:32.343088    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"]
Nov 02 14:08:32.343551 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:32.343347    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" containerID="cri-o://cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca" gracePeriod=30
Nov 02 14:08:36.026574 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:36.026523    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"]
Nov 02 14:08:36.030946 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:36.030908    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5rllj"]
Nov 02 14:08:36.981442 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:36.981408    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:08:36.981630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:36.981467    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:08:37.543996 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.543960    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67eb721a-16f5-4d13-b219-23c56707ad93" path="/var/lib/kubelet/pods/67eb721a-16f5-4d13-b219-23c56707ad93/volumes"
Nov 02 14:08:37.600852 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.600828    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:08:37.696821 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.696745    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d46816-9d54-4328-830a-d497d35d593a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d46816-9d54-4328-830a-d497d35d593a-istiod-ca-cert\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.696821 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.696785    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-data\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.696821    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-kube-api-access-rcxjt\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-kube-api-access-rcxjt\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.696855    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19d46816-9d54-4328-830a-d497d35d593a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d46816-9d54-4328-830a-d497d35d593a-istio-podinfo\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.696906    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-istio-token\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-istio-token\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.696950    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-envoy\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.696979    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-credential-socket\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697043 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697009    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-socket\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697069    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-certs\") pod \"19d46816-9d54-4328-830a-d497d35d593a\" (UID: \"19d46816-9d54-4328-830a-d497d35d593a\") "
Nov 02 14:08:37.697369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697120    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-data" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:37.697369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697150    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d46816-9d54-4328-830a-d497d35d593a-istiod-ca-cert" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:37.697369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697338    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-credential-socket" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:37.697620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697393    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-socket" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:37.697620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697542    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d46816-9d54-4328-830a-d497d35d593a-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.697620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697591    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.697620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697606    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.697620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697602    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-certs" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:37.698075 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.697628    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.699375 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.699345    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-envoy" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:37.699461 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.699369    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19d46816-9d54-4328-830a-d497d35d593a-istio-podinfo" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:37.699461 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.699413    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-kube-api-access-rcxjt" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "kube-api-access-rcxjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:37.699461 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.699432    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-istio-token" pod "19d46816-9d54-4328-830a-d497d35d593a" (UID: "19d46816-9d54-4328-830a-d497d35d593a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:37.798551 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.798510    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxjt\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-kube-api-access-rcxjt\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.798551 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.798542    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d46816-9d54-4328-830a-d497d35d593a-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.798551 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.798551    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d46816-9d54-4328-830a-d497d35d593a-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.798833 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.798595    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.798833 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.798604    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d46816-9d54-4328-830a-d497d35d593a-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:37.805316 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.805286    5572 generic.go:356] "Generic (PLEG): container finished" podID="19d46816-9d54-4328-830a-d497d35d593a" containerID="cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca" exitCode=0
Nov 02 14:08:37.805482 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.805342    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"
Nov 02 14:08:37.805482 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.805371    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" event={"ID":"19d46816-9d54-4328-830a-d497d35d593a","Type":"ContainerDied","Data":"cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca"}
Nov 02 14:08:37.805482 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.805424    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9" event={"ID":"19d46816-9d54-4328-830a-d497d35d593a","Type":"ContainerDied","Data":"6ba1c6be0719046d2cf6876aa6850a1fa0561121da485891081574b6ec1f607a"}
Nov 02 14:08:37.805482 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.805446    5572 scope.go:122] "RemoveContainer" containerID="cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca"
Nov 02 14:08:37.815205 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.815187    5572 scope.go:122] "RemoveContainer" containerID="41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514"
Nov 02 14:08:37.827103 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.827076    5572 scope.go:122] "RemoveContainer" containerID="cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca"
Nov 02 14:08:37.827654 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:08:37.827626    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca\": container with ID starting with cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca not found: ID does not exist" containerID="cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca"
Nov 02 14:08:37.827759 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.827662    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca"} err="failed to get container status \"cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca\": rpc error: code = NotFound desc = could not find container \"cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca\": container with ID starting with cb1f328bebedd80739be8e5a9e52dd54d2280117f03a864c13a921e6b0e089ca not found: ID does not exist"
Nov 02 14:08:37.827759 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.827679    5572 scope.go:122] "RemoveContainer" containerID="41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514"
Nov 02 14:08:37.827990 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:08:37.827975    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514\": container with ID starting with 41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514 not found: ID does not exist" containerID="41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514"
Nov 02 14:08:37.828046 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.827995    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514"} err="failed to get container status \"41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514\": rpc error: code = NotFound desc = could not find container \"41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514\": container with ID starting with 41ea2419f20d4a6fee1733cfe1bbef96e8b65833f96a263a07ab269ecc52d514 not found: ID does not exist"
Nov 02 14:08:37.829988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.829967    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"]
Nov 02 14:08:37.831613 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:37.831595    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0c51da-openshift-default-54c7f5966f4nvb9"]
Nov 02 14:08:39.329529 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:39.329494    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" podUID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:08:39.330004 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:39.329501    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" podUID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:08:39.542143 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:39.542107    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d46816-9d54-4328-830a-d497d35d593a" path="/var/lib/kubelet/pods/19d46816-9d54-4328-830a-d497d35d593a/volumes"
Nov 02 14:08:44.933487 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:44.933456    5572 scope.go:122] "RemoveContainer" containerID="d9ac2b07804fe566979bdbfb9c9d9f8113dfcb4f9e153b7d5bc92fc350808da9"
Nov 02 14:08:45.722523 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.722487    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"]
Nov 02 14:08:45.729388 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.729363    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tl8f9"]
Nov 02 14:08:45.744885 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.744859    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"]
Nov 02 14:08:45.745302 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745289    5572 cpu_manager.go:414] "containerMap: removing container" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy"
Nov 02 14:08:45.745347 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745304    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy"
Nov 02 14:08:45.745347 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745341    5572 cpu_manager.go:414] "containerMap: removing container" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy"
Nov 02 14:08:45.745347 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745347    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy"
Nov 02 14:08:45.745441 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745362    5572 cpu_manager.go:414] "containerMap: removing container" podUID="fec289d0-283d-4da2-becc-8dbb7fd5e5ea" containerName="revoke-keys"
Nov 02 14:08:45.745441 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745368    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec289d0-283d-4da2-becc-8dbb7fd5e5ea" containerName="revoke-keys"
Nov 02 14:08:45.745441 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745433    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy"
Nov 02 14:08:45.745534 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745443    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec289d0-283d-4da2-becc-8dbb7fd5e5ea" containerName="revoke-keys"
Nov 02 14:08:45.745534 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.745451    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d46816-9d54-4328-830a-d497d35d593a" containerName="istio-proxy"
Nov 02 14:08:45.750195 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.750175    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.758692 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.758640    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:08:45.760983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.760961    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"]
Nov 02 14:08:45.761607 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.761583    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:08:45.880657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.880625    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.880844 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.880754    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/47ba56a4-b3ce-4d61-88a9-d61253074b79-kube-api-access-w8rgb\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.880844 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.880786    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.981691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.981554    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.982207 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.981693    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/47ba56a4-b3ce-4d61-88a9-d61253074b79-kube-api-access-w8rgb\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.982207 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.981720    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.982403 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.982380    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.984379 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.984354    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:45.996832 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:45.996808    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/47ba56a4-b3ce-4d61-88a9-d61253074b79-kube-api-access-w8rgb\") pod \"maas-api-revoke-sub-3d4156f1-ck7ht\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:46.067042 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:46.067005    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:46.340445 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:46.340405    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"]
Nov 02 14:08:46.342658 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:08:46.342626    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ba56a4_b3ce_4d61_88a9_d61253074b79.slice/crio-e4f105e4177da54067dfb43be966663f8cc6759eaed3057080c04bd89e894296 WatchSource:0}: Error finding container e4f105e4177da54067dfb43be966663f8cc6759eaed3057080c04bd89e894296: Status 404 returned error can't find the container with id e4f105e4177da54067dfb43be966663f8cc6759eaed3057080c04bd89e894296
Nov 02 14:08:46.852945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:46.852862    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht" event={"ID":"47ba56a4-b3ce-4d61-88a9-d61253074b79","Type":"ContainerStarted","Data":"47dbac4954bc2e11528a1e4d063e89d701b89784cb5abefda6746e8af06a880b"}
Nov 02 14:08:46.852945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:46.852911    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht" event={"ID":"47ba56a4-b3ce-4d61-88a9-d61253074b79","Type":"ContainerStarted","Data":"e4f105e4177da54067dfb43be966663f8cc6759eaed3057080c04bd89e894296"}
Nov 02 14:08:46.872397 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:46.872350    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht" podStartSLOduration=1.872337321 podStartE2EDuration="1.872337321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:46.871232892 +0000 UTC m=+1503.449321757" watchObservedRunningTime="2026-11-02 14:08:46.872337321 +0000 UTC m=+1503.450426188"
Nov 02 14:08:47.022941 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:47.022913    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"]
Nov 02 14:08:47.034287 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:47.034245    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tzvjx"]
Nov 02 14:08:47.541672 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:47.541643    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ab78a8-e4c6-4699-bf52-2f1de35609a8" path="/var/lib/kubelet/pods/62ab78a8-e4c6-4699-bf52-2f1de35609a8/volumes"
Nov 02 14:08:47.542039 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:47.542026    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c134796e-9f06-4068-b8a8-875673cb5570" path="/var/lib/kubelet/pods/c134796e-9f06-4068-b8a8-875673cb5570/volumes"
Nov 02 14:08:47.859456 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:47.859420    5572 generic.go:356] "Generic (PLEG): container finished" podID="47ba56a4-b3ce-4d61-88a9-d61253074b79" containerID="47dbac4954bc2e11528a1e4d063e89d701b89784cb5abefda6746e8af06a880b" exitCode=0
Nov 02 14:08:47.859649 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:47.859513    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht" event={"ID":"47ba56a4-b3ce-4d61-88a9-d61253074b79","Type":"ContainerDied","Data":"47dbac4954bc2e11528a1e4d063e89d701b89784cb5abefda6746e8af06a880b"}
Nov 02 14:08:49.043123 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.043089    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:49.173383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.173355    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:08:49.182597 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.182577    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:08:49.209784 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.209754    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-xks-ca\") pod \"47ba56a4-b3ce-4d61-88a9-d61253074b79\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") "
Nov 02 14:08:49.209938 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.209802    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-openshift-ca\") pod \"47ba56a4-b3ce-4d61-88a9-d61253074b79\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") "
Nov 02 14:08:49.209938 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.209845    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ba56a4-b3ce-4d61-88a9-d61253074b79-kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/47ba56a4-b3ce-4d61-88a9-d61253074b79-kube-api-access-w8rgb\") pod \"47ba56a4-b3ce-4d61-88a9-d61253074b79\" (UID: \"47ba56a4-b3ce-4d61-88a9-d61253074b79\") "
Nov 02 14:08:49.210153 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.210130    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-openshift-ca" pod "47ba56a4-b3ce-4d61-88a9-d61253074b79" (UID: "47ba56a4-b3ce-4d61-88a9-d61253074b79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:49.212886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.212847    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-xks-ca" pod "47ba56a4-b3ce-4d61-88a9-d61253074b79" (UID: "47ba56a4-b3ce-4d61-88a9-d61253074b79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:49.213308 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.213284    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ba56a4-b3ce-4d61-88a9-d61253074b79-kube-api-access-w8rgb" pod "47ba56a4-b3ce-4d61-88a9-d61253074b79" (UID: "47ba56a4-b3ce-4d61-88a9-d61253074b79"). InnerVolumeSpecName "kube-api-access-w8rgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:49.311444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.311403    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66dec9d7-3765-4130-ba40-fc078a2d9b6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dec9d7-3765-4130-ba40-fc078a2d9b6e-plugins-config-volume\") pod \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") "
Nov 02 14:08:49.311653 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.311449    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-kube-api-access-cfp46\" (UniqueName: \"kubernetes.io/projected/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-kube-api-access-cfp46\") pod \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") "
Nov 02 14:08:49.311754 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.311729    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66dec9d7-3765-4130-ba40-fc078a2d9b6e-kube-api-access-rj694\" (UniqueName: \"kubernetes.io/projected/66dec9d7-3765-4130-ba40-fc078a2d9b6e-kube-api-access-rj694\") pod \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\" (UID: \"66dec9d7-3765-4130-ba40-fc078a2d9b6e\") "
Nov 02 14:08:49.311843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.311828    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-plugins-config-volume\") pod \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\" (UID: \"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb\") "
Nov 02 14:08:49.311908 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.311863    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dec9d7-3765-4130-ba40-fc078a2d9b6e-plugins-config-volume" pod "66dec9d7-3765-4130-ba40-fc078a2d9b6e" (UID: "66dec9d7-3765-4130-ba40-fc078a2d9b6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:49.312202 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.312131    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-plugins-config-volume" pod "f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" (UID: "f1b08f34-63d7-4f9c-a7ba-c81bad2094cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:49.312202 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.312135    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:49.312202 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.312174    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ba56a4-b3ce-4d61-88a9-d61253074b79-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:49.312202 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.312191    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rgb\" (UniqueName: \"kubernetes.io/projected/47ba56a4-b3ce-4d61-88a9-d61253074b79-kube-api-access-w8rgb\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:49.312202 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.312202    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dec9d7-3765-4130-ba40-fc078a2d9b6e-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:49.313797 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.313774    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-kube-api-access-cfp46" pod "f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" (UID: "f1b08f34-63d7-4f9c-a7ba-c81bad2094cb"). InnerVolumeSpecName "kube-api-access-cfp46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:49.313910 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.313891    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dec9d7-3765-4130-ba40-fc078a2d9b6e-kube-api-access-rj694" pod "66dec9d7-3765-4130-ba40-fc078a2d9b6e" (UID: "66dec9d7-3765-4130-ba40-fc078a2d9b6e"). InnerVolumeSpecName "kube-api-access-rj694". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:49.413407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.413352    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfp46\" (UniqueName: \"kubernetes.io/projected/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-kube-api-access-cfp46\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:49.413407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.413373    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj694\" (UniqueName: \"kubernetes.io/projected/66dec9d7-3765-4130-ba40-fc078a2d9b6e-kube-api-access-rj694\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:49.413407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.413381    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:08:49.871462 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.871430    5572 generic.go:356] "Generic (PLEG): container finished" podID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" containerID="a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33" exitCode=0
Nov 02 14:08:49.871675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.871494    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"
Nov 02 14:08:49.871675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.871520    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" event={"ID":"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb","Type":"ContainerDied","Data":"a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33"}
Nov 02 14:08:49.871675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.871623    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6" event={"ID":"f1b08f34-63d7-4f9c-a7ba-c81bad2094cb","Type":"ContainerDied","Data":"406a63205e34692702fa4cfd8beaff83b0c53eb738f59aa9fb57f558a67ed00a"}
Nov 02 14:08:49.871675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.871655    5572 scope.go:122] "RemoveContainer" containerID="a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33"
Nov 02 14:08:49.872996 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.872968    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"
Nov 02 14:08:49.872996 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.872977    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht" event={"ID":"47ba56a4-b3ce-4d61-88a9-d61253074b79","Type":"ContainerDied","Data":"e4f105e4177da54067dfb43be966663f8cc6759eaed3057080c04bd89e894296"}
Nov 02 14:08:49.873125 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.873018    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f105e4177da54067dfb43be966663f8cc6759eaed3057080c04bd89e894296"
Nov 02 14:08:49.874418 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.874397    5572 generic.go:356] "Generic (PLEG): container finished" podID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" containerID="b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219" exitCode=0
Nov 02 14:08:49.874549 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.874448    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"
Nov 02 14:08:49.874549 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.874476    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" event={"ID":"66dec9d7-3765-4130-ba40-fc078a2d9b6e","Type":"ContainerDied","Data":"b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219"}
Nov 02 14:08:49.874549 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.874534    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc" event={"ID":"66dec9d7-3765-4130-ba40-fc078a2d9b6e","Type":"ContainerDied","Data":"bc8e025f93620267479b48dac9c8a3ed6fea114331337444b4dbc81db15b6f36"}
Nov 02 14:08:49.881708 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.881692    5572 scope.go:122] "RemoveContainer" containerID="a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33"
Nov 02 14:08:49.881977 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:08:49.881958    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33\": container with ID starting with a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33 not found: ID does not exist" containerID="a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33"
Nov 02 14:08:49.882026 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.881985    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33"} err="failed to get container status \"a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33\": rpc error: code = NotFound desc = could not find container \"a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33\": container with ID starting with a694c3262179053b95a153e81f8e13b264705a47849060cd335124d70d2b7e33 not found: ID does not exist"
Nov 02 14:08:49.882026 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.882000    5572 scope.go:122] "RemoveContainer" containerID="b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219"
Nov 02 14:08:49.889843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.889824    5572 scope.go:122] "RemoveContainer" containerID="b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219"
Nov 02 14:08:49.890090 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:08:49.890072    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219\": container with ID starting with b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219 not found: ID does not exist" containerID="b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219"
Nov 02 14:08:49.890135 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.890097    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219"} err="failed to get container status \"b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219\": rpc error: code = NotFound desc = could not find container \"b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219\": container with ID starting with b7200623817cc4188565608e1774446f30f1058ad148f09d93fa02e9d991a219 not found: ID does not exist"
Nov 02 14:08:49.897999 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.897978    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"]
Nov 02 14:08:49.903828 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.903806    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0c51da-5f87bz7slc"]
Nov 02 14:08:49.917694 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.917668    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"]
Nov 02 14:08:49.924815 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:49.924794    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0c51da-7bc46675-dmmw6"]
Nov 02 14:08:51.429354 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:08:51.429316    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:08:51.541865 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:51.541835    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" path="/var/lib/kubelet/pods/66dec9d7-3765-4130-ba40-fc078a2d9b6e/volumes"
Nov 02 14:08:51.542203 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:08:51.542190    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" path="/var/lib/kubelet/pods/f1b08f34-63d7-4f9c-a7ba-c81bad2094cb/volumes"
Nov 02 14:09:08.055728 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:08.055694    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"]
Nov 02 14:09:08.057881 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:08.057858    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bftvs"]
Nov 02 14:09:09.541241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:09.541210    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15c2749-6586-4947-830f-c669654a31b2" path="/var/lib/kubelet/pods/b15c2749-6586-4947-830f-c669654a31b2/volumes"
Nov 02 14:09:21.429827 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:09:21.429785    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:09:26.027157 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:26.027074    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"]
Nov 02 14:09:26.031013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:26.030987    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m22kg"]
Nov 02 14:09:27.541992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:27.541960    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d098ec-25d9-45c6-8fcd-83d43699d548" path="/var/lib/kubelet/pods/25d098ec-25d9-45c6-8fcd-83d43699d548/volumes"
Nov 02 14:09:44.025596 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:44.025542    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"]
Nov 02 14:09:44.027734 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:44.027710    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j7hfp"]
Nov 02 14:09:44.959577 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:44.959538    5572 scope.go:122] "RemoveContainer" containerID="1d670896a930ccdf41c0ee800945ad24ac47b8ed7560672f2c94a2f9006eda62"
Nov 02 14:09:44.968280 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:44.968260    5572 scope.go:122] "RemoveContainer" containerID="020eb00c0e0d1d11f0b571c6ac93084b558a2ac82f6f8739226b1ff4cb2e4ae1"
Nov 02 14:09:44.976430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:44.976411    5572 scope.go:122] "RemoveContainer" containerID="a03688161ba04d865ebb0caff9c6bac4e50741bbab22d63ecdd9941b60a7b6f1"
Nov 02 14:09:44.984367 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:44.984344    5572 scope.go:122] "RemoveContainer" containerID="24a7fd2f6230328cd6199833eb6d18afc4043a1fee07fb4f1d045fc43383586a"
Nov 02 14:09:45.541422 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:09:45.541390    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab50c40-d66d-4caf-94e4-aba7f1d6ddcf" path="/var/lib/kubelet/pods/aab50c40-d66d-4caf-94e4-aba7f1d6ddcf/volumes"
Nov 02 14:09:51.429744 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:09:51.429699    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:10:11.033658 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:11.033625    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"]
Nov 02 14:10:11.037146 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:11.037123    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wnbnj"]
Nov 02 14:10:11.541305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:11.541261    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e51270-09e2-44ef-9913-8823964b9bf4" path="/var/lib/kubelet/pods/a8e51270-09e2-44ef-9913-8823964b9bf4/volumes"
Nov 02 14:10:21.428614 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:10:21.428576    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:10:36.342666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.342629    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"]
Nov 02 14:10:36.344063 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.344034    5572 cpu_manager.go:414] "containerMap: removing container" podUID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" containerName="payload-pre-processing"
Nov 02 14:10:36.344222 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.344207    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" containerName="payload-pre-processing"
Nov 02 14:10:36.344362 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.344351    5572 cpu_manager.go:414] "containerMap: removing container" podUID="47ba56a4-b3ce-4d61-88a9-d61253074b79" containerName="revoke-keys"
Nov 02 14:10:36.344436 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.344427    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ba56a4-b3ce-4d61-88a9-d61253074b79" containerName="revoke-keys"
Nov 02 14:10:36.344527 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.344518    5572 cpu_manager.go:414] "containerMap: removing container" podUID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" containerName="payload-processing"
Nov 02 14:10:36.344630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.344618    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" containerName="payload-processing"
Nov 02 14:10:36.344921 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.344907    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ba56a4-b3ce-4d61-88a9-d61253074b79" containerName="revoke-keys"
Nov 02 14:10:36.345027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.345018    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b08f34-63d7-4f9c-a7ba-c81bad2094cb" containerName="payload-processing"
Nov 02 14:10:36.345121 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.345111    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="66dec9d7-3765-4130-ba40-fc078a2d9b6e" containerName="payload-pre-processing"
Nov 02 14:10:36.349060 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.349034    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.354293 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.354273    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:10:36.354433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.354284    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:10:36.354761 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.354740    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"]
Nov 02 14:10:36.440688 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.440658    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2992z\" (UniqueName: \"kubernetes.io/projected/84498eee-75ae-49d5-bdd3-07fa32566fc3-kube-api-access-2992z\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.440834 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.440709    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.440834 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.440773    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.541226 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.541191    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.541226 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.541227    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.541429 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.541300    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2992z\" (UniqueName: \"kubernetes.io/projected/84498eee-75ae-49d5-bdd3-07fa32566fc3-kube-api-access-2992z\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.541858 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.541834    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.543677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.543657    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.553914 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.553893    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2992z\" (UniqueName: \"kubernetes.io/projected/84498eee-75ae-49d5-bdd3-07fa32566fc3-kube-api-access-2992z\") pod \"maas-api-revoke-sub-d576e68b-8xlkg\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.664262 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.664163    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:36.940393 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:36.940317    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"]
Nov 02 14:10:36.943420 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:10:36.943383    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84498eee_75ae_49d5_bdd3_07fa32566fc3.slice/crio-68eeffbace6d716f45ee68fb965c7dd360640ee4db6808a563e781f148a5daba WatchSource:0}: Error finding container 68eeffbace6d716f45ee68fb965c7dd360640ee4db6808a563e781f148a5daba: Status 404 returned error can't find the container with id 68eeffbace6d716f45ee68fb965c7dd360640ee4db6808a563e781f148a5daba
Nov 02 14:10:37.434763 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:37.434726    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg" event={"ID":"84498eee-75ae-49d5-bdd3-07fa32566fc3","Type":"ContainerStarted","Data":"f14894dd8f1e9e25b2d943e641be473c7843ccd01839dceaeaba69d1ac8ab8f2"}
Nov 02 14:10:37.435263 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:37.434787    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg" event={"ID":"84498eee-75ae-49d5-bdd3-07fa32566fc3","Type":"ContainerStarted","Data":"68eeffbace6d716f45ee68fb965c7dd360640ee4db6808a563e781f148a5daba"}
Nov 02 14:10:37.455941 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:37.455892    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg" podStartSLOduration=1.455878046 podStartE2EDuration="1.455878046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:37.455081045 +0000 UTC m=+1614.033169904" watchObservedRunningTime="2026-11-02 14:10:37.455878046 +0000 UTC m=+1614.033966912"
Nov 02 14:10:38.442102 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:38.442059    5572 generic.go:356] "Generic (PLEG): container finished" podID="84498eee-75ae-49d5-bdd3-07fa32566fc3" containerID="f14894dd8f1e9e25b2d943e641be473c7843ccd01839dceaeaba69d1ac8ab8f2" exitCode=0
Nov 02 14:10:38.442531 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:38.442140    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg" event={"ID":"84498eee-75ae-49d5-bdd3-07fa32566fc3","Type":"ContainerDied","Data":"f14894dd8f1e9e25b2d943e641be473c7843ccd01839dceaeaba69d1ac8ab8f2"}
Nov 02 14:10:39.578203 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.578179    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:39.672238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.672211    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-openshift-ca\") pod \"84498eee-75ae-49d5-bdd3-07fa32566fc3\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") "
Nov 02 14:10:39.672435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.672310    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84498eee-75ae-49d5-bdd3-07fa32566fc3-kube-api-access-2992z\" (UniqueName: \"kubernetes.io/projected/84498eee-75ae-49d5-bdd3-07fa32566fc3-kube-api-access-2992z\") pod \"84498eee-75ae-49d5-bdd3-07fa32566fc3\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") "
Nov 02 14:10:39.672435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.672373    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-xks-ca\") pod \"84498eee-75ae-49d5-bdd3-07fa32566fc3\" (UID: \"84498eee-75ae-49d5-bdd3-07fa32566fc3\") "
Nov 02 14:10:39.672680 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.672653    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-openshift-ca" pod "84498eee-75ae-49d5-bdd3-07fa32566fc3" (UID: "84498eee-75ae-49d5-bdd3-07fa32566fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:39.673019 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.672998    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:10:39.674644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.674619    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-xks-ca" pod "84498eee-75ae-49d5-bdd3-07fa32566fc3" (UID: "84498eee-75ae-49d5-bdd3-07fa32566fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:10:39.674736 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.674695    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84498eee-75ae-49d5-bdd3-07fa32566fc3-kube-api-access-2992z" pod "84498eee-75ae-49d5-bdd3-07fa32566fc3" (UID: "84498eee-75ae-49d5-bdd3-07fa32566fc3"). InnerVolumeSpecName "kube-api-access-2992z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:39.773886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.773832    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2992z\" (UniqueName: \"kubernetes.io/projected/84498eee-75ae-49d5-bdd3-07fa32566fc3-kube-api-access-2992z\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:10:39.773886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:39.773855    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84498eee-75ae-49d5-bdd3-07fa32566fc3-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:10:40.453337 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:40.453307    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"
Nov 02 14:10:40.453337 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:40.453321    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg" event={"ID":"84498eee-75ae-49d5-bdd3-07fa32566fc3","Type":"ContainerDied","Data":"68eeffbace6d716f45ee68fb965c7dd360640ee4db6808a563e781f148a5daba"}
Nov 02 14:10:40.453548 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:40.453362    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68eeffbace6d716f45ee68fb965c7dd360640ee4db6808a563e781f148a5daba"
Nov 02 14:10:45.025608 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:45.025548    5572 scope.go:122] "RemoveContainer" containerID="59607cffccff882c25425224bf966210699adda0562e7c0c30df79263a374526"
Nov 02 14:10:45.034492 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:45.034471    5572 scope.go:122] "RemoveContainer" containerID="5a810c934d6693c2b10a38eaf43374f2f8d024edf22bc245de9c2c4bf8cbe6d1"
Nov 02 14:10:45.042387 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:45.042367    5572 scope.go:122] "RemoveContainer" containerID="0910a4a9d1ab8ca8b5cf3ef2f509e2239765ba8578bf79749c1083b8761c7ff9"
Nov 02 14:10:51.469434 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:10:51.469394    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:10:55.247971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.247869    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"]
Nov 02 14:10:55.248547 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.248450    5572 cpu_manager.go:414] "containerMap: removing container" podUID="84498eee-75ae-49d5-bdd3-07fa32566fc3" containerName="revoke-keys"
Nov 02 14:10:55.248547 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.248467    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84498eee-75ae-49d5-bdd3-07fa32566fc3" containerName="revoke-keys"
Nov 02 14:10:55.248547 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.248529    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="84498eee-75ae-49d5-bdd3-07fa32566fc3" containerName="revoke-keys"
Nov 02 14:10:55.253516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.253492    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.257708 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.257680    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:10:55.259150 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.259126    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:10:55.261137 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.261117    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"]
Nov 02 14:10:55.321537 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.321505    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnpq\" (UniqueName: \"kubernetes.io/projected/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-kube-api-access-qxnpq\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.321717 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.321704    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.321776 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.321742    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.422445 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.422409    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.422662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.422455    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnpq\" (UniqueName: \"kubernetes.io/projected/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-kube-api-access-qxnpq\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.422662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.422531    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.423142 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.423116    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.424950 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.424929    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.434953 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.434933    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnpq\" (UniqueName: \"kubernetes.io/projected/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-kube-api-access-qxnpq\") pod \"maas-api-revoke-sub-d2bc93bc-nfmzz\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.568995 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.568973    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:55.851124 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:55.850947    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"]
Nov 02 14:10:55.853490 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:10:55.853460    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f969f8_4109_4de9_aa7b_24b1bf1d6ce6.slice/crio-1e9944d8b68fafa67d864100a8af08948f252ff7e8d0cc604477a434bba17b07 WatchSource:0}: Error finding container 1e9944d8b68fafa67d864100a8af08948f252ff7e8d0cc604477a434bba17b07: Status 404 returned error can't find the container with id 1e9944d8b68fafa67d864100a8af08948f252ff7e8d0cc604477a434bba17b07
Nov 02 14:10:56.564388 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:56.564353    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz" event={"ID":"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6","Type":"ContainerStarted","Data":"402bed27cd5b5ab90527fe84b58532931dbec63f9cc395395269838646ef9cc9"}
Nov 02 14:10:56.564798 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:56.564414    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz" event={"ID":"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6","Type":"ContainerStarted","Data":"1e9944d8b68fafa67d864100a8af08948f252ff7e8d0cc604477a434bba17b07"}
Nov 02 14:10:56.584523 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:56.584464    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz" podStartSLOduration=1.5844509100000002 podStartE2EDuration="1.58445091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:56.583331979 +0000 UTC m=+1633.161420857" watchObservedRunningTime="2026-11-02 14:10:56.58445091 +0000 UTC m=+1633.162539776"
Nov 02 14:10:57.570921 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:57.570885    5572 generic.go:356] "Generic (PLEG): container finished" podID="01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" containerID="402bed27cd5b5ab90527fe84b58532931dbec63f9cc395395269838646ef9cc9" exitCode=0
Nov 02 14:10:57.571318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:57.570963    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz" event={"ID":"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6","Type":"ContainerDied","Data":"402bed27cd5b5ab90527fe84b58532931dbec63f9cc395395269838646ef9cc9"}
Nov 02 14:10:58.712532 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.712504    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:58.751240 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.751213    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-kube-api-access-qxnpq\" (UniqueName: \"kubernetes.io/projected/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-kube-api-access-qxnpq\") pod \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") "
Nov 02 14:10:58.751389 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.751265    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-xks-ca\") pod \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") "
Nov 02 14:10:58.751448 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.751385    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-openshift-ca\") pod \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\" (UID: \"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6\") "
Nov 02 14:10:58.751711 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.751690    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-openshift-ca" pod "01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" (UID: "01f969f8-4109-4de9-aa7b-24b1bf1d6ce6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:58.753377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.753350    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-kube-api-access-qxnpq" pod "01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" (UID: "01f969f8-4109-4de9-aa7b-24b1bf1d6ce6"). InnerVolumeSpecName "kube-api-access-qxnpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:58.753487 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.753424    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-xks-ca" pod "01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" (UID: "01f969f8-4109-4de9-aa7b-24b1bf1d6ce6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:10:58.852980 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.852901    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnpq\" (UniqueName: \"kubernetes.io/projected/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-kube-api-access-qxnpq\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:10:58.852980 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.852929    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:10:58.852980 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:58.852940    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:10:59.581728 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:59.581694    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"
Nov 02 14:10:59.581884 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:59.581699    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz" event={"ID":"01f969f8-4109-4de9-aa7b-24b1bf1d6ce6","Type":"ContainerDied","Data":"1e9944d8b68fafa67d864100a8af08948f252ff7e8d0cc604477a434bba17b07"}
Nov 02 14:10:59.581884 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:10:59.581819    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e9944d8b68fafa67d864100a8af08948f252ff7e8d0cc604477a434bba17b07"
Nov 02 14:11:07.201160 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.201127    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"]
Nov 02 14:11:07.201717 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.201580    5572 cpu_manager.go:414] "containerMap: removing container" podUID="01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" containerName="revoke-keys"
Nov 02 14:11:07.201717 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.201594    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" containerName="revoke-keys"
Nov 02 14:11:07.201717 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.201697    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" containerName="revoke-keys"
Nov 02 14:11:07.204877 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.204857    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.209903 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.209876    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7346d7-openshift-default-dockercfg-shdnd\""
Nov 02 14:11:07.225866 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.225844    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"]
Nov 02 14:11:07.233462 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233439    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istiod-ca-cert\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.233638 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233618    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-socket\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.233718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233677    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-envoy\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.233781 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233720    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-certs\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.233781 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233769    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-podinfo\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.233890 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233794    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb2fk\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-kube-api-access-kb2fk\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.233890 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233810    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-data\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.233890 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233839    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-credential-socket\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.234021 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.233951    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-token\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.335475 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335438    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-envoy\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.335723 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335494    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-certs\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.335723 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335520    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-podinfo\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.335723 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335549    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2fk\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-kube-api-access-kb2fk\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.335723 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335589    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-data\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.335723 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335620    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-credential-socket\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.335723 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335670    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-token\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.336069 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335956    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istiod-ca-cert\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.336069 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.335996    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-certs\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.336069 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.336031    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-data\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.336238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.336076    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-credential-socket\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.336238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.336112    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-socket\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.336399 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.336380    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-socket\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.336534 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.336515    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istiod-ca-cert\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.338009 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.337984    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-envoy\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.338259 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.338240    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-podinfo\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.349293 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.349261    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-token\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.349759 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.349742    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2fk\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-kube-api-access-kb2fk\") pod \"e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") " pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.517833 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.517747    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:07.802453 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.802380    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"]
Nov 02 14:11:07.805940 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:11:07.805903    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a0d2f0_0533_4bc0_a520_e757c7d782a1.slice/crio-0c9cd210541a11fdd519cca0ae486f20d86c0f4f792edd559f104b54c4e0912f WatchSource:0}: Error finding container 0c9cd210541a11fdd519cca0ae486f20d86c0f4f792edd559f104b54c4e0912f: Status 404 returned error can't find the container with id 0c9cd210541a11fdd519cca0ae486f20d86c0f4f792edd559f104b54c4e0912f
Nov 02 14:11:07.808109 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.808076    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:11:07.808243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.808157    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:11:07.808243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:07.808203    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:11:08.627186 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:08.627145    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" event={"ID":"51a0d2f0-0533-4bc0-a520-e757c7d782a1","Type":"ContainerStarted","Data":"2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c"}
Nov 02 14:11:08.627602 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:08.627202    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" event={"ID":"51a0d2f0-0533-4bc0-a520-e757c7d782a1","Type":"ContainerStarted","Data":"0c9cd210541a11fdd519cca0ae486f20d86c0f4f792edd559f104b54c4e0912f"}
Nov 02 14:11:08.650595 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:08.650518    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podStartSLOduration=1.650500353 podStartE2EDuration="1.650500353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:08.650089944 +0000 UTC m=+1645.228178822" watchObservedRunningTime="2026-11-02 14:11:08.650500353 +0000 UTC m=+1645.228589218"
Nov 02 14:11:09.518235 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:09.518201    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:09.523272 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:09.523247    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:09.632321 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:09.632288    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:09.633092 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:09.633075    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:12.293465 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.293426    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"]
Nov 02 14:11:12.296939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.296922    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.304387 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.304240    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9e8a234c-dockercfg-sp5js\""
Nov 02 14:11:12.304387 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.304369    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7346d7\""
Nov 02 14:11:12.310918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.310895    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"]
Nov 02 14:11:12.333060 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.333024    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"]
Nov 02 14:11:12.336606 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.336590    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.349330 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.349309    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"]
Nov 02 14:11:12.386092 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.386068    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5ms\" (UniqueName: \"kubernetes.io/projected/11389136-a95f-4a57-898e-1475ecd9fce5-kube-api-access-hs5ms\") pod \"payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.386219 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.386113    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2796e67d-edba-471b-a5e4-2288328ea12c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.386274 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.386254    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkjf\" (UniqueName: \"kubernetes.io/projected/2796e67d-edba-471b-a5e4-2288328ea12c-kube-api-access-fjkjf\") pod \"payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.386313 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.386290    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11389136-a95f-4a57-898e-1475ecd9fce5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.487358 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.487304    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkjf\" (UniqueName: \"kubernetes.io/projected/2796e67d-edba-471b-a5e4-2288328ea12c-kube-api-access-fjkjf\") pod \"payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.487657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.487363    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11389136-a95f-4a57-898e-1475ecd9fce5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.487657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.487464    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5ms\" (UniqueName: \"kubernetes.io/projected/11389136-a95f-4a57-898e-1475ecd9fce5-kube-api-access-hs5ms\") pod \"payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.487657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.487514    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2796e67d-edba-471b-a5e4-2288328ea12c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.488665 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.488641    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11389136-a95f-4a57-898e-1475ecd9fce5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.489539 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.488550    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2796e67d-edba-471b-a5e4-2288328ea12c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.504039 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.504008    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkjf\" (UniqueName: \"kubernetes.io/projected/2796e67d-edba-471b-a5e4-2288328ea12c-kube-api-access-fjkjf\") pod \"payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.509575 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.509523    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5ms\" (UniqueName: \"kubernetes.io/projected/11389136-a95f-4a57-898e-1475ecd9fce5-kube-api-access-hs5ms\") pod \"payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.610768 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.610742    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:12.649376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.649352    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:12.890145 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.890120    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"]
Nov 02 14:11:12.892090 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:11:12.892057    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11389136_a95f_4a57_898e_1475ecd9fce5.slice/crio-bd7e0535bee4e47e851d72b0f75da1aabe27e56eda748de8d556f8b27b7f4a78 WatchSource:0}: Error finding container bd7e0535bee4e47e851d72b0f75da1aabe27e56eda748de8d556f8b27b7f4a78: Status 404 returned error can't find the container with id bd7e0535bee4e47e851d72b0f75da1aabe27e56eda748de8d556f8b27b7f4a78
Nov 02 14:11:12.967374 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:12.967351    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"]
Nov 02 14:11:12.968918 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:11:12.968892    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2796e67d_edba_471b_a5e4_2288328ea12c.slice/crio-e46030e19f7be5a171d8899ad83f84ac11a940b90e639e9ef141b69a94322f49 WatchSource:0}: Error finding container e46030e19f7be5a171d8899ad83f84ac11a940b90e639e9ef141b69a94322f49: Status 404 returned error can't find the container with id e46030e19f7be5a171d8899ad83f84ac11a940b90e639e9ef141b69a94322f49
Nov 02 14:11:13.653436 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.653399    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" event={"ID":"11389136-a95f-4a57-898e-1475ecd9fce5","Type":"ContainerStarted","Data":"6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b"}
Nov 02 14:11:13.653927 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.653537    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:13.653927 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.653613    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" event={"ID":"11389136-a95f-4a57-898e-1475ecd9fce5","Type":"ContainerStarted","Data":"bd7e0535bee4e47e851d72b0f75da1aabe27e56eda748de8d556f8b27b7f4a78"}
Nov 02 14:11:13.654957 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.654927    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" event={"ID":"2796e67d-edba-471b-a5e4-2288328ea12c","Type":"ContainerStarted","Data":"44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764"}
Nov 02 14:11:13.655086 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.654978    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" event={"ID":"2796e67d-edba-471b-a5e4-2288328ea12c","Type":"ContainerStarted","Data":"e46030e19f7be5a171d8899ad83f84ac11a940b90e639e9ef141b69a94322f49"}
Nov 02 14:11:13.655086 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.655021    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:13.675902 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.675845    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" podStartSLOduration=1.67582754 podStartE2EDuration="1.67582754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:13.672812696 +0000 UTC m=+1650.250901563" watchObservedRunningTime="2026-11-02 14:11:13.67582754 +0000 UTC m=+1650.253916407"
Nov 02 14:11:13.694658 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:13.694618    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" podStartSLOduration=1.6946064490000001 podStartE2EDuration="1.694606449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:13.693852496 +0000 UTC m=+1650.271941363" watchObservedRunningTime="2026-11-02 14:11:13.694606449 +0000 UTC m=+1650.272695315"
Nov 02 14:11:15.037325 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:15.037291    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"]
Nov 02 14:11:15.048768 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:15.048738    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wskcd"]
Nov 02 14:11:15.550442 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:15.550405    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325aa455-fcd7-4b76-826b-0111452f775f" path="/var/lib/kubelet/pods/325aa455-fcd7-4b76-826b-0111452f775f/volumes"
Nov 02 14:11:21.450354 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:11:21.450295    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:11:24.633811 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.633783    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 14:11:24.634250 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.633841    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 14:11:24.662907 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.662844    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:11:24.663067 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.662919    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:11:24.737411 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.737381    5572 generic.go:356] "Generic (PLEG): container finished" podID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerID="2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c" exitCode=0
Nov 02 14:11:24.737724 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.737532    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" event={"ID":"51a0d2f0-0533-4bc0-a520-e757c7d782a1","Type":"ContainerDied","Data":"2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c"}
Nov 02 14:11:24.737977 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.737959    5572 scope.go:122] "RemoveContainer" containerID="2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c"
Nov 02 14:11:24.739276 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.739245    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:11:24.739383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.739330    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:11:24.739456 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:24.739379    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:11:25.745006 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:25.744970    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" event={"ID":"51a0d2f0-0533-4bc0-a520-e757c7d782a1","Type":"ContainerStarted","Data":"f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2"}
Nov 02 14:11:26.032093 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:26.032000    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"]
Nov 02 14:11:26.034433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:26.034411    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-7xwdt"]
Nov 02 14:11:26.518664 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:26.518631    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:26.518885 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:26.518858    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 14:11:26.518943 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:26.518917    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 14:11:27.518845 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:27.518810    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 14:11:27.519232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:27.518863    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 14:11:27.540848 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:27.540819    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4478ff7c-9698-4d24-84a1-f7c96e91903d" path="/var/lib/kubelet/pods/4478ff7c-9698-4d24-84a1-f7c96e91903d/volumes"
Nov 02 14:11:28.519102 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:28.519066    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 14:11:28.519526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:28.519123    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 14:11:29.519143 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:29.519109    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 14:11:29.519696 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:29.519168    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 14:11:30.518397 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:30.518362    5572 patch_prober.go:29] interesting pod/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 14:11:30.518586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:30.518417    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 14:11:31.523213 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:31.523186    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:31.523727 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:31.523465    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:31.524277 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:31.524260    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:11:45.082938 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:45.082901    5572 scope.go:122] "RemoveContainer" containerID="7e21cc7c5cf91e5f06cd3b59dec89193e472c73e5f9cdc24c22d52e3e7573288"
Nov 02 14:11:45.098923 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:45.098900    5572 scope.go:122] "RemoveContainer" containerID="11f6501b11877bb111068be4f677a912cde969546fff58dba281120fe0c325c5"
Nov 02 14:11:45.106721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:11:45.106694    5572 scope.go:122] "RemoveContainer" containerID="2559f63961634508f21ff93830ca8c3d5817ec3070d5116e69f4dc5a5bf99a1b"
Nov 02 14:11:51.430461 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:11:51.430425    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:12:10.118996 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.118966    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"]
Nov 02 14:12:10.122607 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.122588    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.130116 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.130087    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:10.130259 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.130153    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:12:10.132535 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.132513    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"]
Nov 02 14:12:10.207735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.207705    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.207895 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.207743    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6xzs\" (UniqueName: \"kubernetes.io/projected/d452288e-bd2e-40ca-b201-e17985be9e13-kube-api-access-c6xzs\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.207895 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.207764    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.308797 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.308761    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.308980 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.308808    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6xzs\" (UniqueName: \"kubernetes.io/projected/d452288e-bd2e-40ca-b201-e17985be9e13-kube-api-access-c6xzs\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.308980 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.308841    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.309508 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.309484    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.311253 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.311233    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.323874 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.323851    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6xzs\" (UniqueName: \"kubernetes.io/projected/d452288e-bd2e-40ca-b201-e17985be9e13-kube-api-access-c6xzs\") pod \"maas-api-revoke-sub-4b252813-qgtgb\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.437010 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.436944    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:10.740505 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:10.740449    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"]
Nov 02 14:12:10.742927 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:12:10.742894    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd452288e_bd2e_40ca_b201_e17985be9e13.slice/crio-012e9d2f96984c1dac1c1eba23d5caac6bec6173d9c02f074e39af63d6f101bf WatchSource:0}: Error finding container 012e9d2f96984c1dac1c1eba23d5caac6bec6173d9c02f074e39af63d6f101bf: Status 404 returned error can't find the container with id 012e9d2f96984c1dac1c1eba23d5caac6bec6173d9c02f074e39af63d6f101bf
Nov 02 14:12:11.013119 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:11.013032    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb" event={"ID":"d452288e-bd2e-40ca-b201-e17985be9e13","Type":"ContainerStarted","Data":"6be1adf627d693f519245dca9843764216b78de9a96a76bc2eccb46fbd4914fd"}
Nov 02 14:12:11.013119 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:11.013093    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb" event={"ID":"d452288e-bd2e-40ca-b201-e17985be9e13","Type":"ContainerStarted","Data":"012e9d2f96984c1dac1c1eba23d5caac6bec6173d9c02f074e39af63d6f101bf"}
Nov 02 14:12:11.045398 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:11.045344    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb" podStartSLOduration=1.045330527 podStartE2EDuration="1.045330527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:11.043480866 +0000 UTC m=+1707.621569735" watchObservedRunningTime="2026-11-02 14:12:11.045330527 +0000 UTC m=+1707.623419414"
Nov 02 14:12:12.019120 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:12.019087    5572 generic.go:356] "Generic (PLEG): container finished" podID="d452288e-bd2e-40ca-b201-e17985be9e13" containerID="6be1adf627d693f519245dca9843764216b78de9a96a76bc2eccb46fbd4914fd" exitCode=0
Nov 02 14:12:12.019515 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:12.019129    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb" event={"ID":"d452288e-bd2e-40ca-b201-e17985be9e13","Type":"ContainerDied","Data":"6be1adf627d693f519245dca9843764216b78de9a96a76bc2eccb46fbd4914fd"}
Nov 02 14:12:13.160656 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.160627    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:13.236096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.236064    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d452288e-bd2e-40ca-b201-e17985be9e13-kube-api-access-c6xzs\" (UniqueName: \"kubernetes.io/projected/d452288e-bd2e-40ca-b201-e17985be9e13-kube-api-access-c6xzs\") pod \"d452288e-bd2e-40ca-b201-e17985be9e13\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") "
Nov 02 14:12:13.236256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.236105    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-openshift-ca\") pod \"d452288e-bd2e-40ca-b201-e17985be9e13\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") "
Nov 02 14:12:13.236256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.236162    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-xks-ca\") pod \"d452288e-bd2e-40ca-b201-e17985be9e13\" (UID: \"d452288e-bd2e-40ca-b201-e17985be9e13\") "
Nov 02 14:12:13.236610 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.236533    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-openshift-ca" pod "d452288e-bd2e-40ca-b201-e17985be9e13" (UID: "d452288e-bd2e-40ca-b201-e17985be9e13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:13.238281 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.238257    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d452288e-bd2e-40ca-b201-e17985be9e13-kube-api-access-c6xzs" pod "d452288e-bd2e-40ca-b201-e17985be9e13" (UID: "d452288e-bd2e-40ca-b201-e17985be9e13"). InnerVolumeSpecName "kube-api-access-c6xzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:13.238367 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.238325    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-xks-ca" pod "d452288e-bd2e-40ca-b201-e17985be9e13" (UID: "d452288e-bd2e-40ca-b201-e17985be9e13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:13.337201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.337173    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6xzs\" (UniqueName: \"kubernetes.io/projected/d452288e-bd2e-40ca-b201-e17985be9e13-kube-api-access-c6xzs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:13.337201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.337197    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:13.337201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:13.337208    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d452288e-bd2e-40ca-b201-e17985be9e13-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:14.029968 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:14.029934    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"
Nov 02 14:12:14.030152 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:14.029968    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb" event={"ID":"d452288e-bd2e-40ca-b201-e17985be9e13","Type":"ContainerDied","Data":"012e9d2f96984c1dac1c1eba23d5caac6bec6173d9c02f074e39af63d6f101bf"}
Nov 02 14:12:14.030152 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:14.030023    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012e9d2f96984c1dac1c1eba23d5caac6bec6173d9c02f074e39af63d6f101bf"
Nov 02 14:12:21.041624 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.041553    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"]
Nov 02 14:12:21.042231 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.042209    5572 cpu_manager.go:414] "containerMap: removing container" podUID="d452288e-bd2e-40ca-b201-e17985be9e13" containerName="revoke-keys"
Nov 02 14:12:21.042319 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.042234    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d452288e-bd2e-40ca-b201-e17985be9e13" containerName="revoke-keys"
Nov 02 14:12:21.042391 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.042377    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d452288e-bd2e-40ca-b201-e17985be9e13" containerName="revoke-keys"
Nov 02 14:12:21.045715 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.045693    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.055420 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.055393    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"]
Nov 02 14:12:21.134360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.134325    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:21.134543 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.134398    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:12:21.229633 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.229590    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.229785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.229712    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.229863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.229844    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhqn\" (UniqueName: \"kubernetes.io/projected/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-kube-api-access-kwhqn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.331228 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.331194    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhqn\" (UniqueName: \"kubernetes.io/projected/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-kube-api-access-kwhqn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.331382 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.331249    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.331382 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.331284    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.331910 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.331889    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.333786 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.333764    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.343741 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.343717    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhqn\" (UniqueName: \"kubernetes.io/projected/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-kube-api-access-kwhqn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.360683 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.360656    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:21.439445 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:12:21.439305    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:12:21.644420 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:21.644354    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"]
Nov 02 14:12:21.647288 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:12:21.647258    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d8b151_3b8f_4679_aeb2_3280e3aa5a8f.slice/crio-90a2f338e6b4e0367f53569bf4dbce0bf9425dd429c69b64154fafd99793907b WatchSource:0}: Error finding container 90a2f338e6b4e0367f53569bf4dbce0bf9425dd429c69b64154fafd99793907b: Status 404 returned error can't find the container with id 90a2f338e6b4e0367f53569bf4dbce0bf9425dd429c69b64154fafd99793907b
Nov 02 14:12:22.071343 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:22.071309    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q" event={"ID":"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f","Type":"ContainerStarted","Data":"00aca135f1d079072d4ed71ad1c91ae3e05cf278a328440bee9b192388a64c97"}
Nov 02 14:12:22.071773 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:22.071375    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q" event={"ID":"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f","Type":"ContainerStarted","Data":"90a2f338e6b4e0367f53569bf4dbce0bf9425dd429c69b64154fafd99793907b"}
Nov 02 14:12:22.096460 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:22.096386    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q" podStartSLOduration=1.096368526 podStartE2EDuration="1.096368526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:22.091481029 +0000 UTC m=+1718.669569896" watchObservedRunningTime="2026-11-02 14:12:22.096368526 +0000 UTC m=+1718.674457395"
Nov 02 14:12:23.078114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:23.078041    5572 generic.go:356] "Generic (PLEG): container finished" podID="62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" containerID="00aca135f1d079072d4ed71ad1c91ae3e05cf278a328440bee9b192388a64c97" exitCode=0
Nov 02 14:12:23.078114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:23.078109    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q" event={"ID":"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f","Type":"ContainerDied","Data":"00aca135f1d079072d4ed71ad1c91ae3e05cf278a328440bee9b192388a64c97"}
Nov 02 14:12:24.222209 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.222185    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:24.359287 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.359210    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-openshift-ca\") pod \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") "
Nov 02 14:12:24.359430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.359283    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-xks-ca\") pod \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") "
Nov 02 14:12:24.359430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.359351    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-kube-api-access-kwhqn\" (UniqueName: \"kubernetes.io/projected/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-kube-api-access-kwhqn\") pod \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\" (UID: \"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f\") "
Nov 02 14:12:24.359651 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.359629    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-openshift-ca" pod "62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" (UID: "62d8b151-3b8f-4679-aeb2-3280e3aa5a8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:24.359738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.359726    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:24.361767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.361738    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-xks-ca" pod "62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" (UID: "62d8b151-3b8f-4679-aeb2-3280e3aa5a8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:24.361767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.361736    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-kube-api-access-kwhqn" pod "62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" (UID: "62d8b151-3b8f-4679-aeb2-3280e3aa5a8f"). InnerVolumeSpecName "kube-api-access-kwhqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:24.460666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.460636    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:24.460666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:24.460662    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwhqn\" (UniqueName: \"kubernetes.io/projected/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f-kube-api-access-kwhqn\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:25.095838 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:25.095801    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q" event={"ID":"62d8b151-3b8f-4679-aeb2-3280e3aa5a8f","Type":"ContainerDied","Data":"90a2f338e6b4e0367f53569bf4dbce0bf9425dd429c69b64154fafd99793907b"}
Nov 02 14:12:25.095838 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:25.095830    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"
Nov 02 14:12:25.096054 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:25.095849    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a2f338e6b4e0367f53569bf4dbce0bf9425dd429c69b64154fafd99793907b"
Nov 02 14:12:31.154817 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:31.154784    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"]
Nov 02 14:12:31.168298 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:31.168262    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"]
Nov 02 14:12:32.119785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:32.119745    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"]
Nov 02 14:12:32.124097 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:32.124070    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7346d7-vms9q"]
Nov 02 14:12:33.541417 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:33.541378    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" path="/var/lib/kubelet/pods/62d8b151-3b8f-4679-aeb2-3280e3aa5a8f/volumes"
Nov 02 14:12:36.155384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:36.155323    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" podUID="2796e67d-edba-471b-a5e4-2288328ea12c" containerName="payload-processing" containerID="cri-o://44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764" gracePeriod=25
Nov 02 14:12:36.169519 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:36.169486    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" podUID="11389136-a95f-4a57-898e-1475ecd9fce5" containerName="payload-pre-processing" containerID="cri-o://6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b" gracePeriod=25
Nov 02 14:12:39.272820 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:39.272744    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"]
Nov 02 14:12:39.273549 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:39.273519    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy" containerID="cri-o://f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2" gracePeriod=30
Nov 02 14:12:44.661283 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.661249    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" podUID="11389136-a95f-4a57-898e-1475ecd9fce5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:12:44.661689 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.661260    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" podUID="2796e67d-edba-471b-a5e4-2288328ea12c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:12:44.927768 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.927746    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:12:44.956822 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.956792    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istiod-ca-cert\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.956951 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.956853    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-envoy\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.956951 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.956883    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-kube-api-access-kb2fk\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-kube-api-access-kb2fk\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.956951 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.956945    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-certs\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.957125 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.956979    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-socket\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.957125 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957050    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-data\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.957125 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957085    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-credential-socket\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.957351 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957159    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-podinfo\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.957351 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957162    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istiod-ca-cert" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:44.957351 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957219    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-token\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-token\") pod \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\" (UID: \"51a0d2f0-0533-4bc0-a520-e757c7d782a1\") "
Nov 02 14:12:44.957351 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957317    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-socket" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:44.957611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957499    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-data" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:44.957678 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957653    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-certs" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:44.957734 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957673    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-credential-socket" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:44.957948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957826    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:44.957948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957847    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:44.957948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957863    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:44.957948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957875    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:44.957948 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.957888    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:44.959490 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.959468    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-envoy" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:44.960088 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.960071    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-token" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:44.960325 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.960297    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-podinfo" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:12:44.960416 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:44.960325    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-kube-api-access-kb2fk" pod "51a0d2f0-0533-4bc0-a520-e757c7d782a1" (UID: "51a0d2f0-0533-4bc0-a520-e757c7d782a1"). InnerVolumeSpecName "kube-api-access-kb2fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:45.059309 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.059282    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:45.059309 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.059310    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:45.059462 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.059319    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0d2f0-0533-4bc0-a520-e757c7d782a1-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:45.059462 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.059328    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb2fk\" (UniqueName: \"kubernetes.io/projected/51a0d2f0-0533-4bc0-a520-e757c7d782a1-kube-api-access-kb2fk\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:45.136458 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.136427    5572 scope.go:122] "RemoveContainer" containerID="3d83709630b2ea1166c2d8519b8c175db7e231badde0cef7a11e0c82f48c6b8b"
Nov 02 14:12:45.145133 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.145115    5572 scope.go:122] "RemoveContainer" containerID="559506a93133f9775ce47626fa1ddf40db7a1bbc524f428c4217b018ff4c7106"
Nov 02 14:12:45.204929 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.204901    5572 generic.go:356] "Generic (PLEG): container finished" podID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerID="f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2" exitCode=0
Nov 02 14:12:45.205053 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.204981    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"
Nov 02 14:12:45.205096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.204981    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" event={"ID":"51a0d2f0-0533-4bc0-a520-e757c7d782a1","Type":"ContainerDied","Data":"f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2"}
Nov 02 14:12:45.205132 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.205117    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf" event={"ID":"51a0d2f0-0533-4bc0-a520-e757c7d782a1","Type":"ContainerDied","Data":"0c9cd210541a11fdd519cca0ae486f20d86c0f4f792edd559f104b54c4e0912f"}
Nov 02 14:12:45.205165 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.205146    5572 scope.go:122] "RemoveContainer" containerID="f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2"
Nov 02 14:12:45.213899 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.213881    5572 scope.go:122] "RemoveContainer" containerID="2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c"
Nov 02 14:12:45.222212 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.222195    5572 scope.go:122] "RemoveContainer" containerID="f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2"
Nov 02 14:12:45.222493 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:12:45.222473    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2\": container with ID starting with f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2 not found: ID does not exist" containerID="f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2"
Nov 02 14:12:45.222627 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.222506    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2"} err="failed to get container status \"f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2\": rpc error: code = NotFound desc = could not find container \"f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2\": container with ID starting with f481119d2ab955b54d5d9e004d869a646e8635ef0d1d2cac5f6e7294bbcca6d2 not found: ID does not exist"
Nov 02 14:12:45.222627 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.222529    5572 scope.go:122] "RemoveContainer" containerID="2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c"
Nov 02 14:12:45.222838 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:12:45.222817    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c\": container with ID starting with 2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c not found: ID does not exist" containerID="2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c"
Nov 02 14:12:45.222893 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.222846    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c"} err="failed to get container status \"2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c\": rpc error: code = NotFound desc = could not find container \"2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c\": container with ID starting with 2bec5556a612f849e09506a57bfb60be608841dc8042fed29a59135ef704e78c not found: ID does not exist"
Nov 02 14:12:45.231902 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.231881    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"]
Nov 02 14:12:45.241070 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.241047    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7346d7-openshift-default-8bb659bfc-4wbsf"]
Nov 02 14:12:45.541466 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:45.541376    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" path="/var/lib/kubelet/pods/51a0d2f0-0533-4bc0-a520-e757c7d782a1/volumes"
Nov 02 14:12:51.431989 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:12:51.431952    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:12:54.661617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:54.661546    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" podUID="11389136-a95f-4a57-898e-1475ecd9fce5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:12:54.661617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:54.661592    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" podUID="2796e67d-edba-471b-a5e4-2288328ea12c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:12:56.419366 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.419311    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:12:56.440253 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.440229    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:12:56.463749 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.463715    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2796e67d-edba-471b-a5e4-2288328ea12c-kube-api-access-fjkjf\" (UniqueName: \"kubernetes.io/projected/2796e67d-edba-471b-a5e4-2288328ea12c-kube-api-access-fjkjf\") pod \"2796e67d-edba-471b-a5e4-2288328ea12c\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") "
Nov 02 14:12:56.463886 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.463794    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2796e67d-edba-471b-a5e4-2288328ea12c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2796e67d-edba-471b-a5e4-2288328ea12c-plugins-config-volume\") pod \"2796e67d-edba-471b-a5e4-2288328ea12c\" (UID: \"2796e67d-edba-471b-a5e4-2288328ea12c\") "
Nov 02 14:12:56.464300 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.464271    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2796e67d-edba-471b-a5e4-2288328ea12c-plugins-config-volume" pod "2796e67d-edba-471b-a5e4-2288328ea12c" (UID: "2796e67d-edba-471b-a5e4-2288328ea12c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:56.466521 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.466496    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2796e67d-edba-471b-a5e4-2288328ea12c-kube-api-access-fjkjf" pod "2796e67d-edba-471b-a5e4-2288328ea12c" (UID: "2796e67d-edba-471b-a5e4-2288328ea12c"). InnerVolumeSpecName "kube-api-access-fjkjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:56.565270 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.565237    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11389136-a95f-4a57-898e-1475ecd9fce5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11389136-a95f-4a57-898e-1475ecd9fce5-plugins-config-volume\") pod \"11389136-a95f-4a57-898e-1475ecd9fce5\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") "
Nov 02 14:12:56.565473 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.565283    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11389136-a95f-4a57-898e-1475ecd9fce5-kube-api-access-hs5ms\" (UniqueName: \"kubernetes.io/projected/11389136-a95f-4a57-898e-1475ecd9fce5-kube-api-access-hs5ms\") pod \"11389136-a95f-4a57-898e-1475ecd9fce5\" (UID: \"11389136-a95f-4a57-898e-1475ecd9fce5\") "
Nov 02 14:12:56.565547 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.565507    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkjf\" (UniqueName: \"kubernetes.io/projected/2796e67d-edba-471b-a5e4-2288328ea12c-kube-api-access-fjkjf\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:56.565547 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.565525    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2796e67d-edba-471b-a5e4-2288328ea12c-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:56.565765 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.565740    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11389136-a95f-4a57-898e-1475ecd9fce5-plugins-config-volume" pod "11389136-a95f-4a57-898e-1475ecd9fce5" (UID: "11389136-a95f-4a57-898e-1475ecd9fce5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:56.567908 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.567881    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11389136-a95f-4a57-898e-1475ecd9fce5-kube-api-access-hs5ms" pod "11389136-a95f-4a57-898e-1475ecd9fce5" (UID: "11389136-a95f-4a57-898e-1475ecd9fce5"). InnerVolumeSpecName "kube-api-access-hs5ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:56.666194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.666168    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11389136-a95f-4a57-898e-1475ecd9fce5-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:56.666194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:56.666188    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5ms\" (UniqueName: \"kubernetes.io/projected/11389136-a95f-4a57-898e-1475ecd9fce5-kube-api-access-hs5ms\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:12:57.269241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.269214    5572 generic.go:356] "Generic (PLEG): container finished" podID="2796e67d-edba-471b-a5e4-2288328ea12c" containerID="44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764" exitCode=0
Nov 02 14:12:57.269381 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.269261    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"
Nov 02 14:12:57.269381 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.269302    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" event={"ID":"2796e67d-edba-471b-a5e4-2288328ea12c","Type":"ContainerDied","Data":"44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764"}
Nov 02 14:12:57.269381 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.269359    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn" event={"ID":"2796e67d-edba-471b-a5e4-2288328ea12c","Type":"ContainerDied","Data":"e46030e19f7be5a171d8899ad83f84ac11a940b90e639e9ef141b69a94322f49"}
Nov 02 14:12:57.269545 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.269388    5572 scope.go:122] "RemoveContainer" containerID="44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764"
Nov 02 14:12:57.270754 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.270727    5572 generic.go:356] "Generic (PLEG): container finished" podID="11389136-a95f-4a57-898e-1475ecd9fce5" containerID="6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b" exitCode=0
Nov 02 14:12:57.270882 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.270775    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"
Nov 02 14:12:57.270882 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.270790    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" event={"ID":"11389136-a95f-4a57-898e-1475ecd9fce5","Type":"ContainerDied","Data":"6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b"}
Nov 02 14:12:57.270882 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.270831    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn" event={"ID":"11389136-a95f-4a57-898e-1475ecd9fce5","Type":"ContainerDied","Data":"bd7e0535bee4e47e851d72b0f75da1aabe27e56eda748de8d556f8b27b7f4a78"}
Nov 02 14:12:57.280405 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.279972    5572 scope.go:122] "RemoveContainer" containerID="44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764"
Nov 02 14:12:57.280405 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:12:57.280313    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764\": container with ID starting with 44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764 not found: ID does not exist" containerID="44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764"
Nov 02 14:12:57.280405 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.280346    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764"} err="failed to get container status \"44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764\": rpc error: code = NotFound desc = could not find container \"44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764\": container with ID starting with 44aef4107b828a472f2e6c763c34574da255c81362a7c41d00f8d8178cf56764 not found: ID does not exist"
Nov 02 14:12:57.280405 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.280369    5572 scope.go:122] "RemoveContainer" containerID="6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b"
Nov 02 14:12:57.288901 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.288876    5572 scope.go:122] "RemoveContainer" containerID="6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b"
Nov 02 14:12:57.289168 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:12:57.289148    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b\": container with ID starting with 6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b not found: ID does not exist" containerID="6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b"
Nov 02 14:12:57.289218 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.289176    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b"} err="failed to get container status \"6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b\": rpc error: code = NotFound desc = could not find container \"6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b\": container with ID starting with 6c79b19d32ca9d771f1753484e2f9f7c68cc019e64a4dcf4271f2c0c135bc30b not found: ID does not exist"
Nov 02 14:12:57.294917 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.294897    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"]
Nov 02 14:12:57.300945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.300922    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7346d7-5cf68b47cdx2rn"]
Nov 02 14:12:57.316029 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.316008    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"]
Nov 02 14:12:57.319194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.319175    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7346d7-54c47zlpnn"]
Nov 02 14:12:57.541156 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.541082    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11389136-a95f-4a57-898e-1475ecd9fce5" path="/var/lib/kubelet/pods/11389136-a95f-4a57-898e-1475ecd9fce5/volumes"
Nov 02 14:12:57.541505 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:12:57.541447    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2796e67d-edba-471b-a5e4-2288328ea12c" path="/var/lib/kubelet/pods/2796e67d-edba-471b-a5e4-2288328ea12c/volumes"
Nov 02 14:13:06.041147 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:06.041111    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"]
Nov 02 14:13:06.043912 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:06.043884    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674c462a-qk69l"]
Nov 02 14:13:07.541165 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:07.541135    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e8a277-5ab3-4b15-b522-e88204253424" path="/var/lib/kubelet/pods/d3e8a277-5ab3-4b15-b522-e88204253424/volumes"
Nov 02 14:13:10.000345 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000309    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"]
Nov 02 14:13:10.000738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000731    5572 cpu_manager.go:414] "containerMap: removing container" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy"
Nov 02 14:13:10.000782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000743    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy"
Nov 02 14:13:10.000782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000762    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2796e67d-edba-471b-a5e4-2288328ea12c" containerName="payload-processing"
Nov 02 14:13:10.000782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000767    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2796e67d-edba-471b-a5e4-2288328ea12c" containerName="payload-processing"
Nov 02 14:13:10.000782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000773    5572 cpu_manager.go:414] "containerMap: removing container" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy"
Nov 02 14:13:10.000782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000778    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000803    5572 cpu_manager.go:414] "containerMap: removing container" podUID="11389136-a95f-4a57-898e-1475ecd9fce5" containerName="payload-pre-processing"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000808    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11389136-a95f-4a57-898e-1475ecd9fce5" containerName="payload-pre-processing"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000815    5572 cpu_manager.go:414] "containerMap: removing container" podUID="62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" containerName="revoke-keys"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000820    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" containerName="revoke-keys"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000879    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2796e67d-edba-471b-a5e4-2288328ea12c" containerName="payload-processing"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000886    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000896    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="11389136-a95f-4a57-898e-1475ecd9fce5" containerName="payload-pre-processing"
Nov 02 14:13:10.000945 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.000902    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d8b151-3b8f-4679-aeb2-3280e3aa5a8f" containerName="revoke-keys"
Nov 02 14:13:10.003793 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.003777    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.014648 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.014621    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-be499ab0-kserve-self-signed-certs\""
Nov 02 14:13:10.015410 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.015383    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"]
Nov 02 14:13:10.124257 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.124218    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-model-cache\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.124434 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.124266    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e50fca9-cdc1-4821-9356-63e00b4c0987-tls-certs\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.124434 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.124343    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-dshm\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.124434 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.124378    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-tmp-dir\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.124595 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.124498    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-home\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.124595 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.124534    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6hx\" (UniqueName: \"kubernetes.io/projected/4e50fca9-cdc1-4821-9356-63e00b4c0987-kube-api-access-sd6hx\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.225992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.225955    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-model-cache\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.225992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226001    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e50fca9-cdc1-4821-9356-63e00b4c0987-tls-certs\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.226213 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226022    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-dshm\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.226213 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226136    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-tmp-dir\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.226295 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226264    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-home\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.226353 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226332    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6hx\" (UniqueName: \"kubernetes.io/projected/4e50fca9-cdc1-4821-9356-63e00b4c0987-kube-api-access-sd6hx\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.226488 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226463    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-model-cache\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.226653 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226533    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-tmp-dir\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.226715 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.226656    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-home\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.228414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.228393    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-dshm\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.228546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.228529    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e50fca9-cdc1-4821-9356-63e00b4c0987-tls-certs\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.238893 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.238870    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6hx\" (UniqueName: \"kubernetes.io/projected/4e50fca9-cdc1-4821-9356-63e00b4c0987-kube-api-access-sd6hx\") pod \"e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") " pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.318520 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.318499    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:10.592587 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:10.592536    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"]
Nov 02 14:13:10.595045 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:13:10.595004    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e50fca9_cdc1_4821_9356_63e00b4c0987.slice/crio-10aa878faebae2bfe48ec908d2c90a5860c4a411fe979b4c4abf2267b65902a0 WatchSource:0}: Error finding container 10aa878faebae2bfe48ec908d2c90a5860c4a411fe979b4c4abf2267b65902a0: Status 404 returned error can't find the container with id 10aa878faebae2bfe48ec908d2c90a5860c4a411fe979b4c4abf2267b65902a0
Nov 02 14:13:11.367861 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:11.367820    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj" event={"ID":"4e50fca9-cdc1-4821-9356-63e00b4c0987","Type":"ContainerStarted","Data":"5e5963e7447723ded7f5b54b9bb628a79f002689dcd473680beff6730999d9c1"}
Nov 02 14:13:11.367861 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:11.367862    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj" event={"ID":"4e50fca9-cdc1-4821-9356-63e00b4c0987","Type":"ContainerStarted","Data":"10aa878faebae2bfe48ec908d2c90a5860c4a411fe979b4c4abf2267b65902a0"}
Nov 02 14:13:11.389354 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:11.389313    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj" podStartSLOduration=2.389300695 podStartE2EDuration="2.389300695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:11.388054379 +0000 UTC m=+1767.966143246" watchObservedRunningTime="2026-11-02 14:13:11.389300695 +0000 UTC m=+1767.967389565"
Nov 02 14:13:12.318906 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:12.318865    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:20.318767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:20.318718    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:20.331440 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:20.331415    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:20.438343 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:20.438320    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:21.429446 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:13:21.429409    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:13:26.853913 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.853879    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"]
Nov 02 14:13:26.854463 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.854450    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a0d2f0-0533-4bc0-a520-e757c7d782a1" containerName="istio-proxy"
Nov 02 14:13:26.857675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.857651    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:26.864259 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.864223    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:13:26.866024 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.865998    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:13:26.867020 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.866987    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"]
Nov 02 14:13:26.988815 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.988780    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:26.989004 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.988891    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2wt\" (UniqueName: \"kubernetes.io/projected/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-kube-api-access-2b2wt\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:26.989004 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:26.988930    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.090414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.090380    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.090630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.090476    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.090630 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.090551    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2wt\" (UniqueName: \"kubernetes.io/projected/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-kube-api-access-2b2wt\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.091276 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.091194    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.092999 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.092978    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.102738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.102718    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2wt\" (UniqueName: \"kubernetes.io/projected/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-kube-api-access-2b2wt\") pod \"maas-api-revoke-sub-452b6d7b-t2bmc\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.172404 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.172315    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:27.445027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.444992    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"]
Nov 02 14:13:27.447612 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:13:27.447582    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cbe482_abe8_4ef0_aa6f_b2c4087b4e68.slice/crio-95699f17de629d3509367b5555152b2f628eb43510d69bf08a2b2b2c9a9b4e36 WatchSource:0}: Error finding container 95699f17de629d3509367b5555152b2f628eb43510d69bf08a2b2b2c9a9b4e36: Status 404 returned error can't find the container with id 95699f17de629d3509367b5555152b2f628eb43510d69bf08a2b2b2c9a9b4e36
Nov 02 14:13:27.461721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:27.461698    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc" event={"ID":"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68","Type":"ContainerStarted","Data":"95699f17de629d3509367b5555152b2f628eb43510d69bf08a2b2b2c9a9b4e36"}
Nov 02 14:13:28.469988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:28.469886    5572 generic.go:356] "Generic (PLEG): container finished" podID="65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" containerID="b372a3ca498ab38252c46aae410ebc674dcbad2b7eb6d370e6e4934aac2807e9" exitCode=0
Nov 02 14:13:28.469988 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:28.469955    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc" event={"ID":"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68","Type":"ContainerDied","Data":"b372a3ca498ab38252c46aae410ebc674dcbad2b7eb6d370e6e4934aac2807e9"}
Nov 02 14:13:29.859264 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:29.859237    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:29.954038 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:29.953993    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-xks-ca\") pod \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") "
Nov 02 14:13:29.954415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:29.954389    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-openshift-ca\") pod \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") "
Nov 02 14:13:29.954691 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:29.954673    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-kube-api-access-2b2wt\" (UniqueName: \"kubernetes.io/projected/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-kube-api-access-2b2wt\") pod \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\" (UID: \"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68\") "
Nov 02 14:13:29.956103 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:29.956077    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-openshift-ca" pod "65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" (UID: "65cbe482-abe8-4ef0-aa6f-b2c4087b4e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:29.965303 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:29.965275    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-xks-ca" pod "65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" (UID: "65cbe482-abe8-4ef0-aa6f-b2c4087b4e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:29.967663 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:29.967626    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-kube-api-access-2b2wt" pod "65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" (UID: "65cbe482-abe8-4ef0-aa6f-b2c4087b4e68"). InnerVolumeSpecName "kube-api-access-2b2wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:30.056400 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:30.056307    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b2wt\" (UniqueName: \"kubernetes.io/projected/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-kube-api-access-2b2wt\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:30.056663 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:30.056642    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:30.056789 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:30.056773    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:30.517586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:30.502435    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc" event={"ID":"65cbe482-abe8-4ef0-aa6f-b2c4087b4e68","Type":"ContainerDied","Data":"95699f17de629d3509367b5555152b2f628eb43510d69bf08a2b2b2c9a9b4e36"}
Nov 02 14:13:30.517586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:30.502503    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95699f17de629d3509367b5555152b2f628eb43510d69bf08a2b2b2c9a9b4e36"
Nov 02 14:13:30.517586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:30.502607    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"
Nov 02 14:13:39.487867 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:39.487827    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"]
Nov 02 14:13:41.060586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.059759    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"]
Nov 02 14:13:41.060586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.060457    5572 cpu_manager.go:414] "containerMap: removing container" podUID="65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" containerName="revoke-keys"
Nov 02 14:13:41.060586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.060478    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" containerName="revoke-keys"
Nov 02 14:13:41.060586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.060594    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" containerName="revoke-keys"
Nov 02 14:13:41.064193 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.064173    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.069457 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.069431    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"]
Nov 02 14:13:41.070305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.070281    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:13:41.070451 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.070414    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:13:41.164536 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.164213    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.164536 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.164292    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.164536 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.164345    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dt2n\" (UniqueName: \"kubernetes.io/projected/846a8578-64ec-443b-bf28-2af9dfac289b-kube-api-access-2dt2n\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.265163 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.265119    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.265376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.265187    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.265376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.265232    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dt2n\" (UniqueName: \"kubernetes.io/projected/846a8578-64ec-443b-bf28-2af9dfac289b-kube-api-access-2dt2n\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.266213 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.266185    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.268277 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.268258    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.277943 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.277923    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dt2n\" (UniqueName: \"kubernetes.io/projected/846a8578-64ec-443b-bf28-2af9dfac289b-kube-api-access-2dt2n\") pod \"maas-api-revoke-sub-3e31a506-qhwbm\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.380724 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.380611    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:41.837918 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:41.801657    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"]
Nov 02 14:13:42.252585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:42.243699    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:13:42.252585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:42.243802    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:42.275717 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:42.275684    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:13:42.275862 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:42.275753    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:42.570057 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:42.570023    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm" event={"ID":"846a8578-64ec-443b-bf28-2af9dfac289b","Type":"ContainerStarted","Data":"dc396689e0a9eb32c5ae705f7d2ad5a73071e70ec560fabeed849d3093a36b1e"}
Nov 02 14:13:42.570224 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:42.570075    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm" event={"ID":"846a8578-64ec-443b-bf28-2af9dfac289b","Type":"ContainerStarted","Data":"1debadf4ad963819bc36ff55d9365103050381d3b0ecbe6d3a94f03ddf4b6964"}
Nov 02 14:13:43.579579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:43.579531    5572 generic.go:356] "Generic (PLEG): container finished" podID="846a8578-64ec-443b-bf28-2af9dfac289b" containerID="dc396689e0a9eb32c5ae705f7d2ad5a73071e70ec560fabeed849d3093a36b1e" exitCode=0
Nov 02 14:13:43.579995 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:43.579596    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm" event={"ID":"846a8578-64ec-443b-bf28-2af9dfac289b","Type":"ContainerDied","Data":"dc396689e0a9eb32c5ae705f7d2ad5a73071e70ec560fabeed849d3093a36b1e"}
Nov 02 14:13:44.970339 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:44.970312    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:45.129585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.125961    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846a8578-64ec-443b-bf28-2af9dfac289b-kube-api-access-2dt2n\" (UniqueName: \"kubernetes.io/projected/846a8578-64ec-443b-bf28-2af9dfac289b-kube-api-access-2dt2n\") pod \"846a8578-64ec-443b-bf28-2af9dfac289b\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") "
Nov 02 14:13:45.133678 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.133640    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-xks-ca\") pod \"846a8578-64ec-443b-bf28-2af9dfac289b\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") "
Nov 02 14:13:45.137682 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.137650    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-openshift-ca\") pod \"846a8578-64ec-443b-bf28-2af9dfac289b\" (UID: \"846a8578-64ec-443b-bf28-2af9dfac289b\") "
Nov 02 14:13:45.150797 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.144028    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846a8578-64ec-443b-bf28-2af9dfac289b-kube-api-access-2dt2n" pod "846a8578-64ec-443b-bf28-2af9dfac289b" (UID: "846a8578-64ec-443b-bf28-2af9dfac289b"). InnerVolumeSpecName "kube-api-access-2dt2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:45.150797 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.148285    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-openshift-ca" pod "846a8578-64ec-443b-bf28-2af9dfac289b" (UID: "846a8578-64ec-443b-bf28-2af9dfac289b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:45.151112 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.151063    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-xks-ca" pod "846a8578-64ec-443b-bf28-2af9dfac289b" (UID: "846a8578-64ec-443b-bf28-2af9dfac289b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:45.172198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.172172    5572 scope.go:122] "RemoveContainer" containerID="fab5f898dd69de99defa48200188506249e1b38a959d6b7225019071d5798564"
Nov 02 14:13:45.261013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.242468    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dt2n\" (UniqueName: \"kubernetes.io/projected/846a8578-64ec-443b-bf28-2af9dfac289b-kube-api-access-2dt2n\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:45.261013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.242502    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:45.261013 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.242518    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846a8578-64ec-443b-bf28-2af9dfac289b-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:45.669194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.669158    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm" event={"ID":"846a8578-64ec-443b-bf28-2af9dfac289b","Type":"ContainerDied","Data":"1debadf4ad963819bc36ff55d9365103050381d3b0ecbe6d3a94f03ddf4b6964"}
Nov 02 14:13:45.669340 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.669216    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1debadf4ad963819bc36ff55d9365103050381d3b0ecbe6d3a94f03ddf4b6964"
Nov 02 14:13:45.669340 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.669334    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"
Nov 02 14:13:45.852096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.851683    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:13:45.852096 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:45.851744    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bnw6z" podUID="f7165380-1253-4546-b0bf-f80b2df80754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:49.032492 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:49.032459    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"]
Nov 02 14:13:49.035885 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:49.035863    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ck7ht"]
Nov 02 14:13:49.541782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:49.541753    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ba56a4-b3ce-4d61-88a9-d61253074b79" path="/var/lib/kubelet/pods/47ba56a4-b3ce-4d61-88a9-d61253074b79/volumes"
Nov 02 14:13:51.431658 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:13:51.431611    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:13:54.509552 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.509470    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj" podUID="4e50fca9-cdc1-4821-9356-63e00b4c0987" containerName="main" containerID="cri-o://5e5963e7447723ded7f5b54b9bb628a79f002689dcd473680beff6730999d9c1" gracePeriod=45
Nov 02 14:13:54.751617 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.751583    5572 generic.go:356] "Generic (PLEG): container finished" podID="4e50fca9-cdc1-4821-9356-63e00b4c0987" containerID="5e5963e7447723ded7f5b54b9bb628a79f002689dcd473680beff6730999d9c1" exitCode=0
Nov 02 14:13:54.751771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.751650    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj" event={"ID":"4e50fca9-cdc1-4821-9356-63e00b4c0987","Type":"ContainerDied","Data":"5e5963e7447723ded7f5b54b9bb628a79f002689dcd473680beff6730999d9c1"}
Nov 02 14:13:54.751771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.751704    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj" event={"ID":"4e50fca9-cdc1-4821-9356-63e00b4c0987","Type":"ContainerDied","Data":"10aa878faebae2bfe48ec908d2c90a5860c4a411fe979b4c4abf2267b65902a0"}
Nov 02 14:13:54.751771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.751722    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10aa878faebae2bfe48ec908d2c90a5860c4a411fe979b4c4abf2267b65902a0"
Nov 02 14:13:54.753392 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.753369    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:54.774045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.772900    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-home\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-home\") pod \"4e50fca9-cdc1-4821-9356-63e00b4c0987\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") "
Nov 02 14:13:54.774045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.772957    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e50fca9-cdc1-4821-9356-63e00b4c0987-kube-api-access-sd6hx\" (UniqueName: \"kubernetes.io/projected/4e50fca9-cdc1-4821-9356-63e00b4c0987-kube-api-access-sd6hx\") pod \"4e50fca9-cdc1-4821-9356-63e00b4c0987\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") "
Nov 02 14:13:54.774045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.773031    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e50fca9-cdc1-4821-9356-63e00b4c0987-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e50fca9-cdc1-4821-9356-63e00b4c0987-tls-certs\") pod \"4e50fca9-cdc1-4821-9356-63e00b4c0987\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") "
Nov 02 14:13:54.774045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.773069    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-model-cache\") pod \"4e50fca9-cdc1-4821-9356-63e00b4c0987\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") "
Nov 02 14:13:54.774045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.773128    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-dshm\") pod \"4e50fca9-cdc1-4821-9356-63e00b4c0987\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") "
Nov 02 14:13:54.774045 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.773164    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-tmp-dir\") pod \"4e50fca9-cdc1-4821-9356-63e00b4c0987\" (UID: \"4e50fca9-cdc1-4821-9356-63e00b4c0987\") "
Nov 02 14:13:54.774589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.774503    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-tmp-dir" pod "4e50fca9-cdc1-4821-9356-63e00b4c0987" (UID: "4e50fca9-cdc1-4821-9356-63e00b4c0987"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.774939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.774900    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-home" pod "4e50fca9-cdc1-4821-9356-63e00b4c0987" (UID: "4e50fca9-cdc1-4821-9356-63e00b4c0987"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.778938 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.776782    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-model-cache" pod "4e50fca9-cdc1-4821-9356-63e00b4c0987" (UID: "4e50fca9-cdc1-4821-9356-63e00b4c0987"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.778938 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.777028    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e50fca9-cdc1-4821-9356-63e00b4c0987-tls-certs" pod "4e50fca9-cdc1-4821-9356-63e00b4c0987" (UID: "4e50fca9-cdc1-4821-9356-63e00b4c0987"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:54.778938 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.778381    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e50fca9-cdc1-4821-9356-63e00b4c0987-kube-api-access-sd6hx" pod "4e50fca9-cdc1-4821-9356-63e00b4c0987" (UID: "4e50fca9-cdc1-4821-9356-63e00b4c0987"). InnerVolumeSpecName "kube-api-access-sd6hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:54.779161 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.778997    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-dshm" pod "4e50fca9-cdc1-4821-9356-63e00b4c0987" (UID: "4e50fca9-cdc1-4821-9356-63e00b4c0987"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.874966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.874930    5572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e50fca9-cdc1-4821-9356-63e00b4c0987-tls-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:54.874966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.874958    5572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-model-cache\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:54.874966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.874971    5572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-dshm\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:54.875198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.874979    5572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-tmp-dir\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:54.875198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.874989    5572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e50fca9-cdc1-4821-9356-63e00b4c0987-home\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:54.875198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:54.874999    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd6hx\" (UniqueName: \"kubernetes.io/projected/4e50fca9-cdc1-4821-9356-63e00b4c0987-kube-api-access-sd6hx\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:13:55.758810 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:55.758777    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"
Nov 02 14:13:55.780618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:55.780466    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"]
Nov 02 14:13:55.783383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:55.783287    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-be499ab0-kserve-6cdfbfc5b9-p6bvj"]
Nov 02 14:13:57.243714 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:57.243658    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-drgtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:13:57.244365 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:57.244312    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-drgtc" podUID="b508b5b5-3d14-4bc5-8462-a45ec55b2b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:57.273694 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:57.273673    5572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:13:57.273860 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:57.273843    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-l7sbn" podUID="fc2035df-6646-4894-946b-6b38c5d60f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:57.541669 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:13:57.541583    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e50fca9-cdc1-4821-9356-63e00b4c0987" path="/var/lib/kubelet/pods/4e50fca9-cdc1-4821-9356-63e00b4c0987/volumes"
Nov 02 14:14:01.400668 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.400633    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"]
Nov 02 14:14:01.401127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.401058    5572 cpu_manager.go:414] "containerMap: removing container" podUID="4e50fca9-cdc1-4821-9356-63e00b4c0987" containerName="main"
Nov 02 14:14:01.401127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.401070    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e50fca9-cdc1-4821-9356-63e00b4c0987" containerName="main"
Nov 02 14:14:01.401127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.401099    5572 cpu_manager.go:414] "containerMap: removing container" podUID="846a8578-64ec-443b-bf28-2af9dfac289b" containerName="revoke-keys"
Nov 02 14:14:01.401127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.401104    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846a8578-64ec-443b-bf28-2af9dfac289b" containerName="revoke-keys"
Nov 02 14:14:01.401258 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.401162    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e50fca9-cdc1-4821-9356-63e00b4c0987" containerName="main"
Nov 02 14:14:01.401258 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.401175    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="846a8578-64ec-443b-bf28-2af9dfac289b" containerName="revoke-keys"
Nov 02 14:14:01.405450 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.405429    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.412836 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.412814    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:14:01.412836 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.412829    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:14:01.414769 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.414733    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"]
Nov 02 14:14:01.466119 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.466083    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wx5j\" (UniqueName: \"kubernetes.io/projected/c910f7a0-b1eb-4166-97cd-931842187df6-kube-api-access-4wx5j\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.466250 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.466134    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.466315 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.466272    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.567527 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.567493    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wx5j\" (UniqueName: \"kubernetes.io/projected/c910f7a0-b1eb-4166-97cd-931842187df6-kube-api-access-4wx5j\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.567713 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.567538    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.567713 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.567627    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.568223 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.568193    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.570126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.570102    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.580672 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.580644    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wx5j\" (UniqueName: \"kubernetes.io/projected/c910f7a0-b1eb-4166-97cd-931842187df6-kube-api-access-4wx5j\") pod \"maas-api-revoke-sub-55f42e7d-xbrc4\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.719012 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.718938    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:01.995301 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:01.995279    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"]
Nov 02 14:14:01.997525 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:14:01.997496    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc910f7a0_b1eb_4166_97cd_931842187df6.slice/crio-a5fb42cdbecf8f14ab5d18f18f241256a2d52569f36b8123af8fd307abec3dc4 WatchSource:0}: Error finding container a5fb42cdbecf8f14ab5d18f18f241256a2d52569f36b8123af8fd307abec3dc4: Status 404 returned error can't find the container with id a5fb42cdbecf8f14ab5d18f18f241256a2d52569f36b8123af8fd307abec3dc4
Nov 02 14:14:02.795331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:02.795300    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4" event={"ID":"c910f7a0-b1eb-4166-97cd-931842187df6","Type":"ContainerStarted","Data":"0257f6d682c111d39e77c9921c808b74954f48d9ac5282169594986cadd6f0d2"}
Nov 02 14:14:02.795840 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:02.795359    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4" event={"ID":"c910f7a0-b1eb-4166-97cd-931842187df6","Type":"ContainerStarted","Data":"a5fb42cdbecf8f14ab5d18f18f241256a2d52569f36b8123af8fd307abec3dc4"}
Nov 02 14:14:02.818099 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:02.818050    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4" podStartSLOduration=1.818035251 podStartE2EDuration="1.818035251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:02.815666657 +0000 UTC m=+1819.393755526" watchObservedRunningTime="2026-11-02 14:14:02.818035251 +0000 UTC m=+1819.396124114"
Nov 02 14:14:03.804658 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:03.804618    5572 generic.go:356] "Generic (PLEG): container finished" podID="c910f7a0-b1eb-4166-97cd-931842187df6" containerID="0257f6d682c111d39e77c9921c808b74954f48d9ac5282169594986cadd6f0d2" exitCode=0
Nov 02 14:14:03.805036 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:03.804672    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4" event={"ID":"c910f7a0-b1eb-4166-97cd-931842187df6","Type":"ContainerDied","Data":"0257f6d682c111d39e77c9921c808b74954f48d9ac5282169594986cadd6f0d2"}
Nov 02 14:14:04.945982 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:04.945958    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:05.001520 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.001491    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c910f7a0-b1eb-4166-97cd-931842187df6-kube-api-access-4wx5j\" (UniqueName: \"kubernetes.io/projected/c910f7a0-b1eb-4166-97cd-931842187df6-kube-api-access-4wx5j\") pod \"c910f7a0-b1eb-4166-97cd-931842187df6\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") "
Nov 02 14:14:05.001735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.001545    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-openshift-ca\") pod \"c910f7a0-b1eb-4166-97cd-931842187df6\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") "
Nov 02 14:14:05.001735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.001592    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-xks-ca\") pod \"c910f7a0-b1eb-4166-97cd-931842187df6\" (UID: \"c910f7a0-b1eb-4166-97cd-931842187df6\") "
Nov 02 14:14:05.001966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.001943    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-openshift-ca" pod "c910f7a0-b1eb-4166-97cd-931842187df6" (UID: "c910f7a0-b1eb-4166-97cd-931842187df6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:05.003843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.003811    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-xks-ca" pod "c910f7a0-b1eb-4166-97cd-931842187df6" (UID: "c910f7a0-b1eb-4166-97cd-931842187df6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:14:05.003951 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.003899    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c910f7a0-b1eb-4166-97cd-931842187df6-kube-api-access-4wx5j" pod "c910f7a0-b1eb-4166-97cd-931842187df6" (UID: "c910f7a0-b1eb-4166-97cd-931842187df6"). InnerVolumeSpecName "kube-api-access-4wx5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:05.102971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.102888    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:05.102971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.102927    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c910f7a0-b1eb-4166-97cd-931842187df6-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:05.102971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.102944    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wx5j\" (UniqueName: \"kubernetes.io/projected/c910f7a0-b1eb-4166-97cd-931842187df6-kube-api-access-4wx5j\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:05.819360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.819329    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"
Nov 02 14:14:05.819360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.819341    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4" event={"ID":"c910f7a0-b1eb-4166-97cd-931842187df6","Type":"ContainerDied","Data":"a5fb42cdbecf8f14ab5d18f18f241256a2d52569f36b8123af8fd307abec3dc4"}
Nov 02 14:14:05.819609 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:05.819383    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5fb42cdbecf8f14ab5d18f18f241256a2d52569f36b8123af8fd307abec3dc4"
Nov 02 14:14:20.684444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.684405    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"]
Nov 02 14:14:20.685499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.684966    5572 cpu_manager.go:414] "containerMap: removing container" podUID="c910f7a0-b1eb-4166-97cd-931842187df6" containerName="revoke-keys"
Nov 02 14:14:20.685499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.684988    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c910f7a0-b1eb-4166-97cd-931842187df6" containerName="revoke-keys"
Nov 02 14:14:20.685499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.685130    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c910f7a0-b1eb-4166-97cd-931842187df6" containerName="revoke-keys"
Nov 02 14:14:20.688441 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.688417    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.700301 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.700280    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8ecb8add-openshift-default-dockercfg-8hzjd\""
Nov 02 14:14:20.710853 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.710828    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"]
Nov 02 14:14:20.751057 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751032    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-certs\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751094    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqp72\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-kube-api-access-jqp72\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751152    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istiod-ca-cert\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751287 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751247    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-token\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751287 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751268    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-credential-socket\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751406 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751385    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-data\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751472 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751447    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-podinfo\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751520 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751477    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-envoy\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.751520 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.751511    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-socket\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.852585 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852535    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-certs\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.852788 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852611    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqp72\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-kube-api-access-jqp72\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.852788 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852628    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istiod-ca-cert\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.852788 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852668    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-token\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.852788 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852695    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-credential-socket\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.852788 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852747    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-data\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.852788 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852776    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-podinfo\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.853127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852912    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-envoy\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.853127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852962    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-socket\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.853127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.852991    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-certs\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.853127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.853042    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-data\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.853331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.853180    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-credential-socket\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.853331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.853299    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-socket\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.853435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.853343    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istiod-ca-cert\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.855310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.855285    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-envoy\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.855522 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.855504    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-podinfo\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.865516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.865494    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-token\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:20.865744 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:20.865723    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqp72\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-kube-api-access-jqp72\") pod \"e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") " pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:21.001643 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.001527    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:21.297294 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.297218    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"]
Nov 02 14:14:21.300225 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:14:21.300195    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b56576_b6e2_4a0a_a42d_474c0a8f2821.slice/crio-4a8e0a05ee642dde4f91b24c129be634a3cb4401a7fe873b9214f6615bf5ff89 WatchSource:0}: Error finding container 4a8e0a05ee642dde4f91b24c129be634a3cb4401a7fe873b9214f6615bf5ff89: Status 404 returned error can't find the container with id 4a8e0a05ee642dde4f91b24c129be634a3cb4401a7fe873b9214f6615bf5ff89
Nov 02 14:14:21.302510 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.302475    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:14:21.302581 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.302543    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:14:21.302618 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.302590    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:14:21.429547 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:21.429502    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:14:21.897516 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.897474    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" event={"ID":"93b56576-b6e2-4a0a-a42d-474c0a8f2821","Type":"ContainerStarted","Data":"643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064"}
Nov 02 14:14:21.897941 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.897531    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" event={"ID":"93b56576-b6e2-4a0a-a42d-474c0a8f2821","Type":"ContainerStarted","Data":"4a8e0a05ee642dde4f91b24c129be634a3cb4401a7fe873b9214f6615bf5ff89"}
Nov 02 14:14:21.923510 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:21.923462    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" podStartSLOduration=1.9234500749999999 podStartE2EDuration="1.923450075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:21.922975646 +0000 UTC m=+1838.501064512" watchObservedRunningTime="2026-11-02 14:14:21.923450075 +0000 UTC m=+1838.501538933"
Nov 02 14:14:22.001997 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:22.001972    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:22.007242 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:22.007222    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:22.910042 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:22.910004    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:22.910994 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:22.910966    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:24.853697 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.853662    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"]
Nov 02 14:14:24.859243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.858928    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:24.879240 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.879208    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"]
Nov 02 14:14:24.879426 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.879407    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8ecb8add\""
Nov 02 14:14:24.879589 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.879548    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8ecb8add-dockercfg-c9tpb\""
Nov 02 14:14:24.895073 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.895042    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"]
Nov 02 14:14:24.899271 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.899254    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:24.915358 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.915332    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"]
Nov 02 14:14:24.996210 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.996070    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s5n\" (UniqueName: \"kubernetes.io/projected/b399c13a-f936-4936-bac9-53816874c43a-kube-api-access-p7s5n\") pod \"payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") " pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:24.996698 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.996662    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgh6\" (UniqueName: \"kubernetes.io/projected/40352d47-9400-40cc-a325-63c19649a4d5-kube-api-access-4hgh6\") pod \"payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:24.996944 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.996908    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40352d47-9400-40cc-a325-63c19649a4d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:24.997107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:24.997081    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b399c13a-f936-4936-bac9-53816874c43a-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") " pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:25.098508 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.098477    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40352d47-9400-40cc-a325-63c19649a4d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:25.098726 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.098528    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b399c13a-f936-4936-bac9-53816874c43a-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") " pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:25.098726 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.098659    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s5n\" (UniqueName: \"kubernetes.io/projected/b399c13a-f936-4936-bac9-53816874c43a-kube-api-access-p7s5n\") pod \"payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") " pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:25.098982 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.098958    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgh6\" (UniqueName: \"kubernetes.io/projected/40352d47-9400-40cc-a325-63c19649a4d5-kube-api-access-4hgh6\") pod \"payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:25.099274 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.099255    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40352d47-9400-40cc-a325-63c19649a4d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:25.099326 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.099308    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b399c13a-f936-4936-bac9-53816874c43a-plugins-config-volume\") pod \"payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") " pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:25.111241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.111186    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgh6\" (UniqueName: \"kubernetes.io/projected/40352d47-9400-40cc-a325-63c19649a4d5-kube-api-access-4hgh6\") pod \"payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:25.111331 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.111255    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s5n\" (UniqueName: \"kubernetes.io/projected/b399c13a-f936-4936-bac9-53816874c43a-kube-api-access-p7s5n\") pod \"payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") " pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:25.174284 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.174250    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:25.213752 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.213723    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:25.451026 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.451002    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"]
Nov 02 14:14:25.453089 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:14:25.453061    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb399c13a_f936_4936_bac9_53816874c43a.slice/crio-552b961509f1a74e74492be6c630bb53bbfddb388bb09e786490b65046f0b01d WatchSource:0}: Error finding container 552b961509f1a74e74492be6c630bb53bbfddb388bb09e786490b65046f0b01d: Status 404 returned error can't find the container with id 552b961509f1a74e74492be6c630bb53bbfddb388bb09e786490b65046f0b01d
Nov 02 14:14:25.509928 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.509900    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"]
Nov 02 14:14:25.511520 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:14:25.511488    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40352d47_9400_40cc_a325_63c19649a4d5.slice/crio-57a9e420221d83b0174952ac9662f1ae409b0763022cbddf4bb07b079357dd4a WatchSource:0}: Error finding container 57a9e420221d83b0174952ac9662f1ae409b0763022cbddf4bb07b079357dd4a: Status 404 returned error can't find the container with id 57a9e420221d83b0174952ac9662f1ae409b0763022cbddf4bb07b079357dd4a
Nov 02 14:14:25.927201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.927165    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz" event={"ID":"40352d47-9400-40cc-a325-63c19649a4d5","Type":"ContainerStarted","Data":"af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034"}
Nov 02 14:14:25.927707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.927221    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz" event={"ID":"40352d47-9400-40cc-a325-63c19649a4d5","Type":"ContainerStarted","Data":"57a9e420221d83b0174952ac9662f1ae409b0763022cbddf4bb07b079357dd4a"}
Nov 02 14:14:25.927707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.927361    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:25.928786 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.928763    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq" event={"ID":"b399c13a-f936-4936-bac9-53816874c43a","Type":"ContainerStarted","Data":"40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2"}
Nov 02 14:14:25.928912 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.928799    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq" event={"ID":"b399c13a-f936-4936-bac9-53816874c43a","Type":"ContainerStarted","Data":"552b961509f1a74e74492be6c630bb53bbfddb388bb09e786490b65046f0b01d"}
Nov 02 14:14:25.928973 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.928906    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:25.973113 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.973076    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq" podStartSLOduration=1.9730668420000002 podStartE2EDuration="1.973066842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:25.971457809 +0000 UTC m=+1842.549546677" watchObservedRunningTime="2026-11-02 14:14:25.973066842 +0000 UTC m=+1842.551155708"
Nov 02 14:14:25.974429 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:25.974400    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz" podStartSLOduration=1.974391129 podStartE2EDuration="1.974391129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:25.951145963 +0000 UTC m=+1842.529234830" watchObservedRunningTime="2026-11-02 14:14:25.974391129 +0000 UTC m=+1842.552479995"
Nov 02 14:14:34.089462 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.089429    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"]
Nov 02 14:14:34.093061 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.093039    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.102509 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.102485    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:14:34.103602 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.103580    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jg4tb\""
Nov 02 14:14:34.113193 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.113165    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"]
Nov 02 14:14:34.190098 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.190058    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.190282 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.190198    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgqnq\" (UniqueName: \"kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.190282 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.190249    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.291111 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.291060    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.291290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.291158    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqnq\" (UniqueName: \"kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.291290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.291178    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.292019 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.291979    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.293900 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.293874    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.302015 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:34.301996    5572 projected.go:196] Error preparing data for projected volume kube-api-access-rgqnq for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 14:14:34.302103 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:34.302063    5572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq podName:14a7225a-6eb3-479b-addc-19fb86d06d08 nodeName:}" failed. No retries permitted until 2026-11-02 14:14:34.80204377 +0000 UTC m=+1851.380132629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rgqnq" (UniqueName: "kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq") pod "maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" (UID: "14a7225a-6eb3-479b-addc-19fb86d06d08") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 14:14:34.896893 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.896857    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqnq\" (UniqueName: \"kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:34.909533 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:34.909508    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqnq\" (UniqueName: \"kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq\") pod \"maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:35.007338 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:35.007305    5572 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" secret="" err="secret \"maas-api-cleanup-dockercfg-jg4tb\" not found"
Nov 02 14:14:35.007522 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:35.007365    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:35.311087 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:35.310933    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"]
Nov 02 14:14:35.313498 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:14:35.313472    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a7225a_6eb3_479b_addc_19fb86d06d08.slice/crio-788bccb570f08f52baed4bb8f8e116460a0adcf8f4c2c3881a26d89b9249ea6b WatchSource:0}: Error finding container 788bccb570f08f52baed4bb8f8e116460a0adcf8f4c2c3881a26d89b9249ea6b: Status 404 returned error can't find the container with id 788bccb570f08f52baed4bb8f8e116460a0adcf8f4c2c3881a26d89b9249ea6b
Nov 02 14:14:35.983121 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:35.983028    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" event={"ID":"14a7225a-6eb3-479b-addc-19fb86d06d08","Type":"ContainerStarted","Data":"6d8b1a03af04f9969f4231020213fb0b4a710836a2c4d5b85cf3433625bf96da"}
Nov 02 14:14:35.983121 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:35.983082    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" event={"ID":"14a7225a-6eb3-479b-addc-19fb86d06d08","Type":"ContainerStarted","Data":"788bccb570f08f52baed4bb8f8e116460a0adcf8f4c2c3881a26d89b9249ea6b"}
Nov 02 14:14:35.983350 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:35.983321    5572 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" secret="" err="secret \"maas-api-cleanup-dockercfg-jg4tb\" not found"
Nov 02 14:14:36.002106 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.002061    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" podStartSLOduration=2.002047839 podStartE2EDuration="2.002047839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:36.000711069 +0000 UTC m=+1852.578799980" watchObservedRunningTime="2026-11-02 14:14:36.002047839 +0000 UTC m=+1852.580136708"
Nov 02 14:14:36.938090 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.938063    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:36.938686 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.938139    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:36.990180 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.990148    5572 generic.go:356] "Generic (PLEG): container finished" podID="14a7225a-6eb3-479b-addc-19fb86d06d08" containerID="6d8b1a03af04f9969f4231020213fb0b4a710836a2c4d5b85cf3433625bf96da" exitCode=0
Nov 02 14:14:36.990360 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.990227    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" event={"ID":"14a7225a-6eb3-479b-addc-19fb86d06d08","Type":"ContainerDied","Data":"6d8b1a03af04f9969f4231020213fb0b4a710836a2c4d5b85cf3433625bf96da"}
Nov 02 14:14:36.992310 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.992286    5572 generic.go:356] "Generic (PLEG): container finished" podID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerID="643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064" exitCode=0
Nov 02 14:14:36.992435 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.992364    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" event={"ID":"93b56576-b6e2-4a0a-a42d-474c0a8f2821","Type":"ContainerDied","Data":"643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064"}
Nov 02 14:14:36.992767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.992753    5572 scope.go:122] "RemoveContainer" containerID="643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064"
Nov 02 14:14:36.994075 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.994048    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:14:36.994165 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.994111    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:14:36.994165 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:36.994147    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:14:37.910346 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:37.910317    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:38.015432 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.015378    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" event={"ID":"93b56576-b6e2-4a0a-a42d-474c0a8f2821","Type":"ContainerStarted","Data":"909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e"}
Nov 02 14:14:38.179173 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.179149    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:38.333683 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.333649    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-xks-ca\") pod \"14a7225a-6eb3-479b-addc-19fb86d06d08\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") "
Nov 02 14:14:38.333904 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.333736    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq\" (UniqueName: \"kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq\") pod \"14a7225a-6eb3-479b-addc-19fb86d06d08\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") "
Nov 02 14:14:38.333904 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.333826    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-openshift-ca\") pod \"14a7225a-6eb3-479b-addc-19fb86d06d08\" (UID: \"14a7225a-6eb3-479b-addc-19fb86d06d08\") "
Nov 02 14:14:38.334284 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.334252    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-openshift-ca" pod "14a7225a-6eb3-479b-addc-19fb86d06d08" (UID: "14a7225a-6eb3-479b-addc-19fb86d06d08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:38.336522 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.336495    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-xks-ca" pod "14a7225a-6eb3-479b-addc-19fb86d06d08" (UID: "14a7225a-6eb3-479b-addc-19fb86d06d08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:14:38.336649 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.336522    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq" pod "14a7225a-6eb3-479b-addc-19fb86d06d08" (UID: "14a7225a-6eb3-479b-addc-19fb86d06d08"). InnerVolumeSpecName "kube-api-access-rgqnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:38.436382 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.436231    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:38.436382 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.436269    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgqnq\" (UniqueName: \"kubernetes.io/projected/14a7225a-6eb3-479b-addc-19fb86d06d08-kube-api-access-rgqnq\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:38.436382 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:38.436285    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a7225a-6eb3-479b-addc-19fb86d06d08-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:39.011587 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:39.007082    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:39.011587 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:39.007235    5572 patch_prober.go:29] interesting pod/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:14:39.011587 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:39.007278    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 14:14:39.022500 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:39.022475    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"
Nov 02 14:14:39.022984 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:39.022515    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8" event={"ID":"14a7225a-6eb3-479b-addc-19fb86d06d08","Type":"ContainerDied","Data":"788bccb570f08f52baed4bb8f8e116460a0adcf8f4c2c3881a26d89b9249ea6b"}
Nov 02 14:14:39.022984 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:39.022580    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="788bccb570f08f52baed4bb8f8e116460a0adcf8f4c2c3881a26d89b9249ea6b"
Nov 02 14:14:40.004586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:40.002019    5572 patch_prober.go:29] interesting pod/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:14:40.004586 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:40.002087    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 14:14:41.005605 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:41.002218    5572 patch_prober.go:29] interesting pod/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:14:41.005605 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:41.002300    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 14:14:42.005974 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:42.005940    5572 patch_prober.go:29] interesting pod/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:14:42.006396 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:42.005992    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 14:14:43.008719 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:43.008688    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:43.009321 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:43.009252    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:43.010337 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:43.010314    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:44.393964 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:44.393931    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"]
Nov 02 14:14:44.411413 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:44.411287    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"]
Nov 02 14:14:45.288816 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:45.288780    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"]
Nov 02 14:14:45.291402 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:45.291366    5572 scope.go:122] "RemoveContainer" containerID="bd452a7f0ce9afe88368727f470784844d1e9ea3b8ad9e146a0bf20c19b6d6c4"
Nov 02 14:14:45.292378 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:45.292337    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ecb8add-5dzm8"]
Nov 02 14:14:45.305007 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:45.304978    5572 scope.go:122] "RemoveContainer" containerID="da26de0abe4d3296a379063b49262a5b5f1008c073f41f4470b19c9e59d46116"
Nov 02 14:14:45.541171 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:45.541081    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a7225a-6eb3-479b-addc-19fb86d06d08" path="/var/lib/kubelet/pods/14a7225a-6eb3-479b-addc-19fb86d06d08/volumes"
Nov 02 14:14:49.394862 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.394815    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq" podUID="b399c13a-f936-4936-bac9-53816874c43a" containerName="payload-processing" containerID="cri-o://40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2" gracePeriod=25
Nov 02 14:14:49.412316 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.412272    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz" podUID="40352d47-9400-40cc-a325-63c19649a4d5" containerName="payload-pre-processing" containerID="cri-o://af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034" gracePeriod=25
Nov 02 14:14:49.662141 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.662119    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:49.693997 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.693973    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:49.762042 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.762016    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b399c13a-f936-4936-bac9-53816874c43a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b399c13a-f936-4936-bac9-53816874c43a-plugins-config-volume\") pod \"b399c13a-f936-4936-bac9-53816874c43a\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") "
Nov 02 14:14:49.762169 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.762128    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b399c13a-f936-4936-bac9-53816874c43a-kube-api-access-p7s5n\" (UniqueName: \"kubernetes.io/projected/b399c13a-f936-4936-bac9-53816874c43a-kube-api-access-p7s5n\") pod \"b399c13a-f936-4936-bac9-53816874c43a\" (UID: \"b399c13a-f936-4936-bac9-53816874c43a\") "
Nov 02 14:14:49.762372 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.762352    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b399c13a-f936-4936-bac9-53816874c43a-plugins-config-volume" pod "b399c13a-f936-4936-bac9-53816874c43a" (UID: "b399c13a-f936-4936-bac9-53816874c43a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:49.764296 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.764276    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b399c13a-f936-4936-bac9-53816874c43a-kube-api-access-p7s5n" pod "b399c13a-f936-4936-bac9-53816874c43a" (UID: "b399c13a-f936-4936-bac9-53816874c43a"). InnerVolumeSpecName "kube-api-access-p7s5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.863351 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.863331    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40352d47-9400-40cc-a325-63c19649a4d5-kube-api-access-4hgh6\" (UniqueName: \"kubernetes.io/projected/40352d47-9400-40cc-a325-63c19649a4d5-kube-api-access-4hgh6\") pod \"40352d47-9400-40cc-a325-63c19649a4d5\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") "
Nov 02 14:14:49.863459 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.863378    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40352d47-9400-40cc-a325-63c19649a4d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40352d47-9400-40cc-a325-63c19649a4d5-plugins-config-volume\") pod \"40352d47-9400-40cc-a325-63c19649a4d5\" (UID: \"40352d47-9400-40cc-a325-63c19649a4d5\") "
Nov 02 14:14:49.863816 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.863795    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7s5n\" (UniqueName: \"kubernetes.io/projected/b399c13a-f936-4936-bac9-53816874c43a-kube-api-access-p7s5n\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:49.863903 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.863821    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b399c13a-f936-4936-bac9-53816874c43a-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:49.863903 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.863821    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40352d47-9400-40cc-a325-63c19649a4d5-plugins-config-volume" pod "40352d47-9400-40cc-a325-63c19649a4d5" (UID: "40352d47-9400-40cc-a325-63c19649a4d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:49.865433 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.865412    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40352d47-9400-40cc-a325-63c19649a4d5-kube-api-access-4hgh6" pod "40352d47-9400-40cc-a325-63c19649a4d5" (UID: "40352d47-9400-40cc-a325-63c19649a4d5"). InnerVolumeSpecName "kube-api-access-4hgh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.964819 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.964738    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40352d47-9400-40cc-a325-63c19649a4d5-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:49.964819 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:49.964766    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hgh6\" (UniqueName: \"kubernetes.io/projected/40352d47-9400-40cc-a325-63c19649a4d5-kube-api-access-4hgh6\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:50.080764 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.080733    5572 generic.go:356] "Generic (PLEG): container finished" podID="40352d47-9400-40cc-a325-63c19649a4d5" containerID="af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034" exitCode=0
Nov 02 14:14:50.080909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.080777    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"
Nov 02 14:14:50.080909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.080820    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz" event={"ID":"40352d47-9400-40cc-a325-63c19649a4d5","Type":"ContainerDied","Data":"af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034"}
Nov 02 14:14:50.080909 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.080881    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz" event={"ID":"40352d47-9400-40cc-a325-63c19649a4d5","Type":"ContainerDied","Data":"57a9e420221d83b0174952ac9662f1ae409b0763022cbddf4bb07b079357dd4a"}
Nov 02 14:14:50.081059 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.080913    5572 scope.go:122] "RemoveContainer" containerID="af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034"
Nov 02 14:14:50.082175 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.082150    5572 generic.go:356] "Generic (PLEG): container finished" podID="b399c13a-f936-4936-bac9-53816874c43a" containerID="40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2" exitCode=0
Nov 02 14:14:50.082301 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.082187    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"
Nov 02 14:14:50.082301 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.082254    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq" event={"ID":"b399c13a-f936-4936-bac9-53816874c43a","Type":"ContainerDied","Data":"40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2"}
Nov 02 14:14:50.082301 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.082293    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq" event={"ID":"b399c13a-f936-4936-bac9-53816874c43a","Type":"ContainerDied","Data":"552b961509f1a74e74492be6c630bb53bbfddb388bb09e786490b65046f0b01d"}
Nov 02 14:14:50.091675 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.091653    5572 scope.go:122] "RemoveContainer" containerID="af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034"
Nov 02 14:14:50.096795 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:50.096766    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034\": container with ID starting with af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034 not found: ID does not exist" containerID="af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034"
Nov 02 14:14:50.096969 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.096925    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034"} err="failed to get container status \"af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034\": rpc error: code = NotFound desc = could not find container \"af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034\": container with ID starting with af60d55a72705b79437ab91cb3eca2af6ec89f1c4f30a187b35ab86a91794034 not found: ID does not exist"
Nov 02 14:14:50.097064 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.096971    5572 scope.go:122] "RemoveContainer" containerID="40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2"
Nov 02 14:14:50.106001 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.105982    5572 scope.go:122] "RemoveContainer" containerID="40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2"
Nov 02 14:14:50.106266 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:50.106238    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2\": container with ID starting with 40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2 not found: ID does not exist" containerID="40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2"
Nov 02 14:14:50.106429 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.106274    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2"} err="failed to get container status \"40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2\": rpc error: code = NotFound desc = could not find container \"40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2\": container with ID starting with 40eb0a2c21486b60ba2c65723a915458189d91cc9203a2ac87fd39675bff77b2 not found: ID does not exist"
Nov 02 14:14:50.108444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.108424    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"]
Nov 02 14:14:50.112949 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.112922    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8ecb8add-77976db9c7-6rczq"]
Nov 02 14:14:50.126091 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.126072    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"]
Nov 02 14:14:50.131154 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.131135    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ecb8add-5fc5664687-s2gqz"]
Nov 02 14:14:50.417238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.417210    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"]
Nov 02 14:14:50.417664 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:50.417509    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy" containerID="cri-o://909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e" gracePeriod=30
Nov 02 14:14:51.429452 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:51.429419    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:14:51.541350 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:51.541323    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40352d47-9400-40cc-a325-63c19649a4d5" path="/var/lib/kubelet/pods/40352d47-9400-40cc-a325-63c19649a4d5/volumes"
Nov 02 14:14:51.541738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:51.541724    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b399c13a-f936-4936-bac9-53816874c43a" path="/var/lib/kubelet/pods/b399c13a-f936-4936-bac9-53816874c43a/volumes"
Nov 02 14:14:55.672745 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.672721    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:55.821113 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821083    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-credential-socket\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821282 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821125    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istiod-ca-cert\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821282 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821172    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-certs\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821282 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821245    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-envoy\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821284    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-kube-api-access-jqp72\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-kube-api-access-jqp72\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821316    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-data\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821379    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-podinfo\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821376    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-credential-socket" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.821415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821407    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-token\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-token\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821469    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-socket\") pod \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\" (UID: \"93b56576-b6e2-4a0a-a42d-474c0a8f2821\") "
Nov 02 14:14:55.821735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821515    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-certs" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.821735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821653    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-data" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.821735 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821680    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istiod-ca-cert" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:55.821936 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.821891    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-socket" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.822189 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.822159    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.822189 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.822183    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.822347 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.822197    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.822347 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.822209    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.822347 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.822221    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.823795 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.823772    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-token" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:55.823912 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.823794    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-podinfo" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:55.824145 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.824122    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-kube-api-access-jqp72" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "kube-api-access-jqp72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:55.824145 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.824136    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-envoy" pod "93b56576-b6e2-4a0a-a42d-474c0a8f2821" (UID: "93b56576-b6e2-4a0a-a42d-474c0a8f2821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.923484 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.923460    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.923484 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.923481    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.923642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.923491    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b56576-b6e2-4a0a-a42d-474c0a8f2821-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:55.923642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:55.923499    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqp72\" (UniqueName: \"kubernetes.io/projected/93b56576-b6e2-4a0a-a42d-474c0a8f2821-kube-api-access-jqp72\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:14:56.116327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.116247    5572 generic.go:356] "Generic (PLEG): container finished" podID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerID="909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e" exitCode=0
Nov 02 14:14:56.116327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.116298    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"
Nov 02 14:14:56.116504 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.116330    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" event={"ID":"93b56576-b6e2-4a0a-a42d-474c0a8f2821","Type":"ContainerDied","Data":"909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e"}
Nov 02 14:14:56.116504 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.116378    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv" event={"ID":"93b56576-b6e2-4a0a-a42d-474c0a8f2821","Type":"ContainerDied","Data":"4a8e0a05ee642dde4f91b24c129be634a3cb4401a7fe873b9214f6615bf5ff89"}
Nov 02 14:14:56.116504 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.116399    5572 scope.go:122] "RemoveContainer" containerID="909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e"
Nov 02 14:14:56.129807 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.129778    5572 scope.go:122] "RemoveContainer" containerID="643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064"
Nov 02 14:14:56.137854 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.137837    5572 scope.go:122] "RemoveContainer" containerID="909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e"
Nov 02 14:14:56.138099 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:56.138081    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e\": container with ID starting with 909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e not found: ID does not exist" containerID="909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e"
Nov 02 14:14:56.138168 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.138106    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e"} err="failed to get container status \"909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e\": rpc error: code = NotFound desc = could not find container \"909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e\": container with ID starting with 909b50283ecbb4544a8f8ab2161ff975c9b4036ad9ce7b6e7df7a051b9983e7e not found: ID does not exist"
Nov 02 14:14:56.138168 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.138122    5572 scope.go:122] "RemoveContainer" containerID="643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064"
Nov 02 14:14:56.138373 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:14:56.138357    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064\": container with ID starting with 643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064 not found: ID does not exist" containerID="643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064"
Nov 02 14:14:56.138420 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.138376    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064"} err="failed to get container status \"643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064\": rpc error: code = NotFound desc = could not find container \"643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064\": container with ID starting with 643c1a014a24bbcbb20f0b26c4f12ae9f19060cda5d2f04c28fc41df4a97b064 not found: ID does not exist"
Nov 02 14:14:56.147960 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.147936    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"]
Nov 02 14:14:56.149376 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:56.149355    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8ecb8add-openshift-default-6bbf7f6b4c-j2zwv"]
Nov 02 14:14:57.541966 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:14:57.541938    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" path="/var/lib/kubelet/pods/93b56576-b6e2-4a0a-a42d-474c0a8f2821/volumes"
Nov 02 14:15:00.150657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.150625    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"]
Nov 02 14:15:00.151035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151002    5572 cpu_manager.go:414] "containerMap: removing container" podUID="40352d47-9400-40cc-a325-63c19649a4d5" containerName="payload-pre-processing"
Nov 02 14:15:00.151035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151012    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40352d47-9400-40cc-a325-63c19649a4d5" containerName="payload-pre-processing"
Nov 02 14:15:00.151035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151033    5572 cpu_manager.go:414] "containerMap: removing container" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151038    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151056    5572 cpu_manager.go:414] "containerMap: removing container" podUID="b399c13a-f936-4936-bac9-53816874c43a" containerName="payload-processing"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151061    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b399c13a-f936-4936-bac9-53816874c43a" containerName="payload-processing"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151077    5572 cpu_manager.go:414] "containerMap: removing container" podUID="14a7225a-6eb3-479b-addc-19fb86d06d08" containerName="revoke-keys"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151082    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a7225a-6eb3-479b-addc-19fb86d06d08" containerName="revoke-keys"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151134    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b399c13a-f936-4936-bac9-53816874c43a" containerName="payload-processing"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151145    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151151    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a7225a-6eb3-479b-addc-19fb86d06d08" containerName="revoke-keys"
Nov 02 14:15:00.151188 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.151157    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="40352d47-9400-40cc-a325-63c19649a4d5" containerName="payload-pre-processing"
Nov 02 14:15:00.155760 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.155743    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.161153 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.161129    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-r4m97\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.161248 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.161180    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6tq\" (UniqueName: \"kubernetes.io/projected/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-kube-api-access-zq6tq\") pod \"maas-api-key-cleanup-29893815-r4m97\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.163602 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.163549    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9c58\""
Nov 02 14:15:00.169067 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.169047    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"]
Nov 02 14:15:00.261794 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.261771    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-r4m97\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.261937 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.261814    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6tq\" (UniqueName: \"kubernetes.io/projected/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-kube-api-access-zq6tq\") pod \"maas-api-key-cleanup-29893815-r4m97\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.262405 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.262388    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-r4m97\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.275137 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.275110    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6tq\" (UniqueName: \"kubernetes.io/projected/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-kube-api-access-zq6tq\") pod \"maas-api-key-cleanup-29893815-r4m97\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.468499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.468421    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:00.745074 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:00.745048    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"]
Nov 02 14:15:00.746826 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:15:00.746800    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeb9b33_76d5_4af0_94b0_1bc0ae6f7fe7.slice/crio-a0ebf6625c758341f08468402749ee098237a44f34dd30dc3da56f13fa9db23e WatchSource:0}: Error finding container a0ebf6625c758341f08468402749ee098237a44f34dd30dc3da56f13fa9db23e: Status 404 returned error can't find the container with id a0ebf6625c758341f08468402749ee098237a44f34dd30dc3da56f13fa9db23e
Nov 02 14:15:01.142192 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:01.142161    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" event={"ID":"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7","Type":"ContainerStarted","Data":"3ebce3a94cf5121a6c6c28c685385b564c82e5781486573b6dfcbcc45160f581"}
Nov 02 14:15:01.142306 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:01.142245    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" event={"ID":"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7","Type":"ContainerStarted","Data":"a0ebf6625c758341f08468402749ee098237a44f34dd30dc3da56f13fa9db23e"}
Nov 02 14:15:01.162349 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:01.162304    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" podStartSLOduration=1.162292085 podStartE2EDuration="1.162292085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.160628087 +0000 UTC m=+1877.738716965" watchObservedRunningTime="2026-11-02 14:15:01.162292085 +0000 UTC m=+1877.740380991"
Nov 02 14:15:02.150141 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.150109    5572 generic.go:356] "Generic (PLEG): container finished" podID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerID="3ebce3a94cf5121a6c6c28c685385b564c82e5781486573b6dfcbcc45160f581" exitCode=60
Nov 02 14:15:02.150319 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.150153    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" event={"ID":"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7","Type":"ContainerDied","Data":"3ebce3a94cf5121a6c6c28c685385b564c82e5781486573b6dfcbcc45160f581"}
Nov 02 14:15:02.150451 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.150434    5572 scope.go:122] "RemoveContainer" containerID="3ebce3a94cf5121a6c6c28c685385b564c82e5781486573b6dfcbcc45160f581"
Nov 02 14:15:02.153713 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.153688    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"]
Nov 02 14:15:02.154165 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.154148    5572 cpu_manager.go:414] "containerMap: removing container" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy"
Nov 02 14:15:02.154165 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.154165    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy"
Nov 02 14:15:02.154301 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.154252    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b56576-b6e2-4a0a-a42d-474c0a8f2821" containerName="istio-proxy"
Nov 02 14:15:02.158194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.158176    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.165897 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.165871    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-560eaa-openshift-default-dockercfg-vtjn7\""
Nov 02 14:15:02.179699 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.179639    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"]
Nov 02 14:15:02.184205 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.183978    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4zgp\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-kube-api-access-n4zgp\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184205 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184034    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-certs\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184205 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184158    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-istio-token\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184453 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184228    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-credential-socket\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184453 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184260    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc51337-7116-48ac-af1a-d856a8edde40-istiod-ca-cert\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184453 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184306    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-socket\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184453 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184378    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-envoy\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184453 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184405    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-data\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.184453 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.184448    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc51337-7116-48ac-af1a-d856a8edde40-istio-podinfo\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286104 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286071    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-istio-token\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286112    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-credential-socket\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286141    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc51337-7116-48ac-af1a-d856a8edde40-istiod-ca-cert\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286172    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-socket\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286231    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-envoy\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286252    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-data\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286305 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286288    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc51337-7116-48ac-af1a-d856a8edde40-istio-podinfo\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286690 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286368    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4zgp\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-kube-api-access-n4zgp\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286690 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286404    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-certs\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286690 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286518    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-credential-socket\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.286850 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.286728    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-certs\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.287174 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.287150    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-socket\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.287336 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.287315    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc51337-7116-48ac-af1a-d856a8edde40-istiod-ca-cert\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.287417 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.287337    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-data\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.293985 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.293925    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc51337-7116-48ac-af1a-d856a8edde40-istio-podinfo\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.297081 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.297053    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-envoy\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.314032 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.313227    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-istio-token\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.315900 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.315871    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4zgp\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-kube-api-access-n4zgp\") pod \"e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") " pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.475879 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.475842    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:02.759867 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.759792    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"]
Nov 02 14:15:02.763597 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:15:02.763550    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc51337_7116_48ac_af1a_d856a8edde40.slice/crio-609b7543965ec91182cadf143366805407df046c3389aebd6a897df716244059 WatchSource:0}: Error finding container 609b7543965ec91182cadf143366805407df046c3389aebd6a897df716244059: Status 404 returned error can't find the container with id 609b7543965ec91182cadf143366805407df046c3389aebd6a897df716244059
Nov 02 14:15:02.766025 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.765991    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:15:02.766106 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.766061    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:15:02.766106 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:02.766096    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:15:03.157487 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.157453    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" event={"ID":"edc51337-7116-48ac-af1a-d856a8edde40","Type":"ContainerStarted","Data":"5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d"}
Nov 02 14:15:03.157718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.157512    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" event={"ID":"edc51337-7116-48ac-af1a-d856a8edde40","Type":"ContainerStarted","Data":"609b7543965ec91182cadf143366805407df046c3389aebd6a897df716244059"}
Nov 02 14:15:03.159320 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.159296    5572 generic.go:356] "Generic (PLEG): container finished" podID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerID="33461b074b89b9ab7a0d1756355a01169f68345263a7c2256f7700dcbcb575e8" exitCode=60
Nov 02 14:15:03.159449 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.159369    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" event={"ID":"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7","Type":"ContainerDied","Data":"33461b074b89b9ab7a0d1756355a01169f68345263a7c2256f7700dcbcb575e8"}
Nov 02 14:15:03.159449 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.159411    5572 scope.go:122] "RemoveContainer" containerID="3ebce3a94cf5121a6c6c28c685385b564c82e5781486573b6dfcbcc45160f581"
Nov 02 14:15:03.159608 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.159589    5572 scope.go:122] "RemoveContainer" containerID="33461b074b89b9ab7a0d1756355a01169f68345263a7c2256f7700dcbcb575e8"
Nov 02 14:15:03.159835 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:15:03.159813    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-r4m97_odh-ai-gateway-infra(bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7"
Nov 02 14:15:03.183499 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.183454    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podStartSLOduration=1.183439037 podStartE2EDuration="1.183439037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.181349047 +0000 UTC m=+1879.759437916" watchObservedRunningTime="2026-11-02 14:15:03.183439037 +0000 UTC m=+1879.761527973"
Nov 02 14:15:03.476876 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.476794    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:03.482070 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:03.482048    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:04.166529 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:04.166492    5572 scope.go:122] "RemoveContainer" containerID="33461b074b89b9ab7a0d1756355a01169f68345263a7c2256f7700dcbcb575e8"
Nov 02 14:15:04.166734 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:04.166587    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:04.166826 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:15:04.166793    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-r4m97_odh-ai-gateway-infra(bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7"
Nov 02 14:15:04.167771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:04.167754    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:06.822354 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.822313    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"]
Nov 02 14:15:06.826415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.826393    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:06.844817 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.844799    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-560eaa\""
Nov 02 14:15:06.847489 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.847462    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-560eaa-dockercfg-7mxzp\""
Nov 02 14:15:06.848962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.848937    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"]
Nov 02 14:15:06.875377 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.875350    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"]
Nov 02 14:15:06.879666 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.879648    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:06.900935 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.900907    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"]
Nov 02 14:15:06.940384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.937733    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dbg\" (UniqueName: \"kubernetes.io/projected/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-kube-api-access-v8dbg\") pod \"payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:06.940384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.937864    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wgz\" (UniqueName: \"kubernetes.io/projected/cf53cdb1-5caf-491f-b3ec-c6136f137949-kube-api-access-d6wgz\") pod \"payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:06.940384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.937927    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf53cdb1-5caf-491f-b3ec-c6136f137949-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:06.940384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:06.938033    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:07.038894 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.038855    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dbg\" (UniqueName: \"kubernetes.io/projected/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-kube-api-access-v8dbg\") pod \"payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:07.039107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.038906    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wgz\" (UniqueName: \"kubernetes.io/projected/cf53cdb1-5caf-491f-b3ec-c6136f137949-kube-api-access-d6wgz\") pod \"payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:07.039107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.038949    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf53cdb1-5caf-491f-b3ec-c6136f137949-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:07.039107 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.039019    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:07.039730 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.039706    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf53cdb1-5caf-491f-b3ec-c6136f137949-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:07.039829 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.039748    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:07.052122 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.052096    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wgz\" (UniqueName: \"kubernetes.io/projected/cf53cdb1-5caf-491f-b3ec-c6136f137949-kube-api-access-d6wgz\") pod \"payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:07.052216 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.052096    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dbg\" (UniqueName: \"kubernetes.io/projected/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-kube-api-access-v8dbg\") pod \"payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:07.139704 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.139639    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:07.195829 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.195795    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:07.417014 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.416979    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"]
Nov 02 14:15:07.419370 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:15:07.419340    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf53cdb1_5caf_491f_b3ec_c6136f137949.slice/crio-53581f16a8aa5d0ce52176503e8b8014675b904a8f7d91751dcf041a9ab7aa51 WatchSource:0}: Error finding container 53581f16a8aa5d0ce52176503e8b8014675b904a8f7d91751dcf041a9ab7aa51: Status 404 returned error can't find the container with id 53581f16a8aa5d0ce52176503e8b8014675b904a8f7d91751dcf041a9ab7aa51
Nov 02 14:15:07.480832 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:07.480810    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"]
Nov 02 14:15:07.482423 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:15:07.482387    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd56285b_33f0_4b90_b725_8fbd0b6eefb7.slice/crio-c4c889bfa743693e8aa006638ab529ebcfd0745c050213db5c99bf1aeb263f63 WatchSource:0}: Error finding container c4c889bfa743693e8aa006638ab529ebcfd0745c050213db5c99bf1aeb263f63: Status 404 returned error can't find the container with id c4c889bfa743693e8aa006638ab529ebcfd0745c050213db5c99bf1aeb263f63
Nov 02 14:15:08.188764 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.188724    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26" event={"ID":"cf53cdb1-5caf-491f-b3ec-c6136f137949","Type":"ContainerStarted","Data":"4b49ee2b6528e2ce8c8a55ea3628ecf05067cc5738c5ce55942d2fc51ac9d3a6"}
Nov 02 14:15:08.189224 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.188775    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26" event={"ID":"cf53cdb1-5caf-491f-b3ec-c6136f137949","Type":"ContainerStarted","Data":"53581f16a8aa5d0ce52176503e8b8014675b904a8f7d91751dcf041a9ab7aa51"}
Nov 02 14:15:08.189224 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.188840    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:08.190539 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.190514    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv" event={"ID":"cd56285b-33f0-4b90-b725-8fbd0b6eefb7","Type":"ContainerStarted","Data":"9ce575d2c32b32697ae5f5984ea700af98427638cbfb6a62cc07ea7e1ff634b3"}
Nov 02 14:15:08.190677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.190551    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv" event={"ID":"cd56285b-33f0-4b90-b725-8fbd0b6eefb7","Type":"ContainerStarted","Data":"c4c889bfa743693e8aa006638ab529ebcfd0745c050213db5c99bf1aeb263f63"}
Nov 02 14:15:08.190677 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.190666    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:08.208681 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.208635    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26" podStartSLOduration=2.208622367 podStartE2EDuration="2.208622367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:08.206987979 +0000 UTC m=+1884.785076858" watchObservedRunningTime="2026-11-02 14:15:08.208622367 +0000 UTC m=+1884.786711234"
Nov 02 14:15:08.229853 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:08.229800    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv" podStartSLOduration=2.229783178 podStartE2EDuration="2.229783178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:08.22745692 +0000 UTC m=+1884.805545786" watchObservedRunningTime="2026-11-02 14:15:08.229783178 +0000 UTC m=+1884.807872043"
Nov 02 14:15:16.535761 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:16.535728    5572 scope.go:122] "RemoveContainer" containerID="33461b074b89b9ab7a0d1756355a01169f68345263a7c2256f7700dcbcb575e8"
Nov 02 14:15:17.268232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:17.268201    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" event={"ID":"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7","Type":"ContainerStarted","Data":"8e3a2b0f40200a68a7d22803d90bbf3a1af5439738894586ca9da608608fdb59"}
Nov 02 14:15:18.274613 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:18.274544    5572 generic.go:356] "Generic (PLEG): container finished" podID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerID="8e3a2b0f40200a68a7d22803d90bbf3a1af5439738894586ca9da608608fdb59" exitCode=60
Nov 02 14:15:18.274613 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:18.274603    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" event={"ID":"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7","Type":"ContainerDied","Data":"8e3a2b0f40200a68a7d22803d90bbf3a1af5439738894586ca9da608608fdb59"}
Nov 02 14:15:18.275161 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:18.274670    5572 scope.go:122] "RemoveContainer" containerID="33461b074b89b9ab7a0d1756355a01169f68345263a7c2256f7700dcbcb575e8"
Nov 02 14:15:18.275161 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:18.275013    5572 scope.go:122] "RemoveContainer" containerID="8e3a2b0f40200a68a7d22803d90bbf3a1af5439738894586ca9da608608fdb59"
Nov 02 14:15:18.275309 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:15:18.275285    5572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-r4m97_odh-ai-gateway-infra(bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7"
Nov 02 14:15:18.316248 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:18.316225    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"]
Nov 02 14:15:19.167620 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.167589    5572 patch_prober.go:29] interesting pod/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 14:15:19.167831 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.167645    5572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 14:15:19.200169 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.200143    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:15:19.200334 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.200213    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:15:19.281474 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.281441    5572 generic.go:356] "Generic (PLEG): container finished" podID="edc51337-7116-48ac-af1a-d856a8edde40" containerID="5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d" exitCode=0
Nov 02 14:15:19.281958 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.281519    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" event={"ID":"edc51337-7116-48ac-af1a-d856a8edde40","Type":"ContainerDied","Data":"5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d"}
Nov 02 14:15:19.282040 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.282010    5572 scope.go:122] "RemoveContainer" containerID="5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d"
Nov 02 14:15:19.283480 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.283443    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:15:19.283621 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.283525    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:15:19.283621 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.283601    5572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 14:15:19.476108 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.476073    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:19.604599 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.604549    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-maas-api-service-ca\") pod \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") "
Nov 02 14:15:19.604771 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.604747    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-kube-api-access-zq6tq\" (UniqueName: \"kubernetes.io/projected/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-kube-api-access-zq6tq\") pod \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\" (UID: \"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7\") "
Nov 02 14:15:19.604925 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.604902    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-maas-api-service-ca" pod "bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" (UID: "bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:19.605094 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.605079    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-maas-api-service-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:15:19.607000 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.606975    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-kube-api-access-zq6tq" pod "bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" (UID: "bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7"). InnerVolumeSpecName "kube-api-access-zq6tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:19.706413 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:19.706258    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq6tq\" (UniqueName: \"kubernetes.io/projected/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7-kube-api-access-zq6tq\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:15:20.320611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.320572    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"
Nov 02 14:15:20.320611 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.320587    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97" event={"ID":"bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7","Type":"ContainerDied","Data":"a0ebf6625c758341f08468402749ee098237a44f34dd30dc3da56f13fa9db23e"}
Nov 02 14:15:20.321785 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.320683    5572 scope.go:122] "RemoveContainer" containerID="8e3a2b0f40200a68a7d22803d90bbf3a1af5439738894586ca9da608608fdb59"
Nov 02 14:15:20.323437 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.323409    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" event={"ID":"edc51337-7116-48ac-af1a-d856a8edde40","Type":"ContainerStarted","Data":"31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153"}
Nov 02 14:15:20.407020 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.406973    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"]
Nov 02 14:15:20.408902 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.408877    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-r4m97"]
Nov 02 14:15:20.482027 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.481994    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:20.482393 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.482368    5572 patch_prober.go:29] interesting pod/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 14:15:20.482574 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:20.482530    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 14:15:21.432640 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:15:21.432602    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:15:21.476335 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:21.476307    5572 patch_prober.go:29] interesting pod/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 14:15:21.476470 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:21.476354    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 14:15:21.541362 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:21.541333    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" path="/var/lib/kubelet/pods/bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7/volumes"
Nov 02 14:15:22.476238 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:22.476204    5572 patch_prober.go:29] interesting pod/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 14:15:22.476657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:22.476255    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 14:15:23.476782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:23.476694    5572 patch_prober.go:29] interesting pod/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 14:15:23.476782 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:23.476753    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 14:15:24.477581 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:24.477532    5572 patch_prober.go:29] interesting pod/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 14:15:24.478201 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:24.478167    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 14:15:25.476953 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:25.476911    5572 patch_prober.go:29] interesting pod/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 14:15:25.477146 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:25.476976    5572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 14:15:26.480273 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:26.480242    5572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:26.480682 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:26.480471    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:26.481247 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:26.481228    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:15:40.039251 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:40.039220    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"]
Nov 02 14:15:40.043327 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:40.043303    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8xlkg"]
Nov 02 14:15:41.541684 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:41.541647    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84498eee-75ae-49d5-bdd3-07fa32566fc3" path="/var/lib/kubelet/pods/84498eee-75ae-49d5-bdd3-07fa32566fc3/volumes"
Nov 02 14:15:45.332915 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:45.332875    5572 scope.go:122] "RemoveContainer" containerID="47dbac4954bc2e11528a1e4d063e89d701b89784cb5abefda6746e8af06a880b"
Nov 02 14:15:51.430359 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:15:51.430318    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:15:59.041219 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:59.041175    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"]
Nov 02 14:15:59.053194 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:59.053168    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nfmzz"]
Nov 02 14:15:59.541742 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:15:59.541709    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f969f8-4109-4de9-aa7b-24b1bf1d6ce6" path="/var/lib/kubelet/pods/01f969f8-4109-4de9-aa7b-24b1bf1d6ce6/volumes"
Nov 02 14:16:21.431202 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:16:21.431163    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:16:45.375979 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:45.375945    5572 scope.go:122] "RemoveContainer" containerID="f14894dd8f1e9e25b2d943e641be473c7843ccd01839dceaeaba69d1ac8ab8f2"
Nov 02 14:16:50.532268 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.532229    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"]
Nov 02 14:16:50.532897 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.532876    5572 cpu_manager.go:414] "containerMap: removing container" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:16:50.532993 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.532899    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:16:50.532993 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.532916    5572 cpu_manager.go:414] "containerMap: removing container" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:16:50.532993 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.532924    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:16:50.533156 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.533083    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:16:50.533156 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.533098    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:16:50.537827 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.537791    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.545138 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.545113    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:16:50.545843 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.545760    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9c58\""
Nov 02 14:16:50.546539 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.546510    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"]
Nov 02 14:16:50.634542 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.634510    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.634542 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.634547    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mtbc\" (UniqueName: \"kubernetes.io/projected/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-kube-api-access-6mtbc\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.634794 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.634724    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.736282 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.736243    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.736480 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.736346    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.736480 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.736379    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mtbc\" (UniqueName: \"kubernetes.io/projected/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-kube-api-access-6mtbc\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.736908 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.736886    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.738871 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.738848    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.748910 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.748889    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mtbc\" (UniqueName: \"kubernetes.io/projected/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-kube-api-access-6mtbc\") pod \"maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:50.853439 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:50.853413    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:51.127662 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:51.127619    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"]
Nov 02 14:16:51.129857 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:16:51.129830    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5bc480_40e1_4021_9a5d_bc2e8abfbbac.slice/crio-527be3d8148f25081c29a9147bb2418bd2aaf141c31ab8b79ba16402e4e40ec8 WatchSource:0}: Error finding container 527be3d8148f25081c29a9147bb2418bd2aaf141c31ab8b79ba16402e4e40ec8: Status 404 returned error can't find the container with id 527be3d8148f25081c29a9147bb2418bd2aaf141c31ab8b79ba16402e4e40ec8
Nov 02 14:16:51.437776 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:16:51.437739    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:16:51.799148 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:51.799056    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q" event={"ID":"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac","Type":"ContainerStarted","Data":"47d60c5cc27b06bff410b451104d4d7f0d8fe8d6e4bd39d2319b21c64112d987"}
Nov 02 14:16:51.799148 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:51.799122    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q" event={"ID":"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac","Type":"ContainerStarted","Data":"527be3d8148f25081c29a9147bb2418bd2aaf141c31ab8b79ba16402e4e40ec8"}
Nov 02 14:16:51.818501 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:51.818457    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q" podStartSLOduration=1.8184450970000001 podStartE2EDuration="1.818445097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:51.817437734 +0000 UTC m=+1988.395526602" watchObservedRunningTime="2026-11-02 14:16:51.818445097 +0000 UTC m=+1988.396533963"
Nov 02 14:16:52.805772 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:52.805738    5572 generic.go:356] "Generic (PLEG): container finished" podID="2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" containerID="47d60c5cc27b06bff410b451104d4d7f0d8fe8d6e4bd39d2319b21c64112d987" exitCode=0
Nov 02 14:16:52.806240 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:52.805814    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q" event={"ID":"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac","Type":"ContainerDied","Data":"47d60c5cc27b06bff410b451104d4d7f0d8fe8d6e4bd39d2319b21c64112d987"}
Nov 02 14:16:53.944394 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:53.944369    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:53.971117 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:53.971089    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-xks-ca\") pod \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") "
Nov 02 14:16:53.971278 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:53.971139    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-openshift-ca\") pod \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") "
Nov 02 14:16:53.971452 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:53.971426    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-kube-api-access-6mtbc\" (UniqueName: \"kubernetes.io/projected/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-kube-api-access-6mtbc\") pod \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\" (UID: \"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac\") "
Nov 02 14:16:53.974175 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:53.974144    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-xks-ca" pod "2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" (UID: "2e5bc480-40e1-4021-9a5d-bc2e8abfbbac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:16:53.975711 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:53.975661    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-kube-api-access-6mtbc" pod "2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" (UID: "2e5bc480-40e1-4021-9a5d-bc2e8abfbbac"). InnerVolumeSpecName "kube-api-access-6mtbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:16:53.978930 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:53.978901    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-openshift-ca" pod "2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" (UID: "2e5bc480-40e1-4021-9a5d-bc2e8abfbbac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:16:54.073029 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:54.072955    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:16:54.073029 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:54.072980    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:16:54.073029 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:54.072991    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mtbc\" (UniqueName: \"kubernetes.io/projected/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac-kube-api-access-6mtbc\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:16:54.817866 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:54.817835    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q" event={"ID":"2e5bc480-40e1-4021-9a5d-bc2e8abfbbac","Type":"ContainerDied","Data":"527be3d8148f25081c29a9147bb2418bd2aaf141c31ab8b79ba16402e4e40ec8"}
Nov 02 14:16:54.817866 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:54.817860    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"
Nov 02 14:16:54.818126 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:16:54.817886    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="527be3d8148f25081c29a9147bb2418bd2aaf141c31ab8b79ba16402e4e40ec8"
Nov 02 14:17:00.668939 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:00.668909    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"]
Nov 02 14:17:00.683158 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:00.683112    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"]
Nov 02 14:17:05.669324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.669284    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv" podUID="cd56285b-33f0-4b90-b725-8fbd0b6eefb7" containerName="payload-processing" containerID="cri-o://9ce575d2c32b32697ae5f5984ea700af98427638cbfb6a62cc07ea7e1ff634b3" gracePeriod=25
Nov 02 14:17:05.683851 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.683808    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26" podUID="cf53cdb1-5caf-491f-b3ec-c6136f137949" containerName="payload-pre-processing" containerID="cri-o://4b49ee2b6528e2ce8c8a55ea3628ecf05067cc5738c5ce55942d2fc51ac9d3a6" gracePeriod=25
Nov 02 14:17:05.879348 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.879308    5572 generic.go:356] "Generic (PLEG): container finished" podID="cf53cdb1-5caf-491f-b3ec-c6136f137949" containerID="4b49ee2b6528e2ce8c8a55ea3628ecf05067cc5738c5ce55942d2fc51ac9d3a6" exitCode=0
Nov 02 14:17:05.879529 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.879371    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26" event={"ID":"cf53cdb1-5caf-491f-b3ec-c6136f137949","Type":"ContainerDied","Data":"4b49ee2b6528e2ce8c8a55ea3628ecf05067cc5738c5ce55942d2fc51ac9d3a6"}
Nov 02 14:17:05.881163 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.881141    5572 generic.go:356] "Generic (PLEG): container finished" podID="cd56285b-33f0-4b90-b725-8fbd0b6eefb7" containerID="9ce575d2c32b32697ae5f5984ea700af98427638cbfb6a62cc07ea7e1ff634b3" exitCode=0
Nov 02 14:17:05.881270 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.881190    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv" event={"ID":"cd56285b-33f0-4b90-b725-8fbd0b6eefb7","Type":"ContainerDied","Data":"9ce575d2c32b32697ae5f5984ea700af98427638cbfb6a62cc07ea7e1ff634b3"}
Nov 02 14:17:05.932837 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.932782    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:17:05.959607 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:05.959580    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:17:06.092805 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.092774    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf53cdb1-5caf-491f-b3ec-c6136f137949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf53cdb1-5caf-491f-b3ec-c6136f137949-plugins-config-volume\") pod \"cf53cdb1-5caf-491f-b3ec-c6136f137949\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") "
Nov 02 14:17:06.092962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.092855    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf53cdb1-5caf-491f-b3ec-c6136f137949-kube-api-access-d6wgz\" (UniqueName: \"kubernetes.io/projected/cf53cdb1-5caf-491f-b3ec-c6136f137949-kube-api-access-d6wgz\") pod \"cf53cdb1-5caf-491f-b3ec-c6136f137949\" (UID: \"cf53cdb1-5caf-491f-b3ec-c6136f137949\") "
Nov 02 14:17:06.092962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.092919    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-plugins-config-volume\") pod \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") "
Nov 02 14:17:06.092962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.092942    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-kube-api-access-v8dbg\" (UniqueName: \"kubernetes.io/projected/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-kube-api-access-v8dbg\") pod \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\" (UID: \"cd56285b-33f0-4b90-b725-8fbd0b6eefb7\") "
Nov 02 14:17:06.093311 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.093280    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf53cdb1-5caf-491f-b3ec-c6136f137949-plugins-config-volume" pod "cf53cdb1-5caf-491f-b3ec-c6136f137949" (UID: "cf53cdb1-5caf-491f-b3ec-c6136f137949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:06.093311 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.093288    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-plugins-config-volume" pod "cd56285b-33f0-4b90-b725-8fbd0b6eefb7" (UID: "cd56285b-33f0-4b90-b725-8fbd0b6eefb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:06.095163 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.095144    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf53cdb1-5caf-491f-b3ec-c6136f137949-kube-api-access-d6wgz" pod "cf53cdb1-5caf-491f-b3ec-c6136f137949" (UID: "cf53cdb1-5caf-491f-b3ec-c6136f137949"). InnerVolumeSpecName "kube-api-access-d6wgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:06.095241 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.095196    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-kube-api-access-v8dbg" pod "cd56285b-33f0-4b90-b725-8fbd0b6eefb7" (UID: "cd56285b-33f0-4b90-b725-8fbd0b6eefb7"). InnerVolumeSpecName "kube-api-access-v8dbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:06.194243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.194172    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf53cdb1-5caf-491f-b3ec-c6136f137949-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:06.194243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.194204    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6wgz\" (UniqueName: \"kubernetes.io/projected/cf53cdb1-5caf-491f-b3ec-c6136f137949-kube-api-access-d6wgz\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:06.194243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.194214    5572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-plugins-config-volume\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:06.194243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.194224    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dbg\" (UniqueName: \"kubernetes.io/projected/cd56285b-33f0-4b90-b725-8fbd0b6eefb7-kube-api-access-v8dbg\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:06.417285 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.417250    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"]
Nov 02 14:17:06.419859 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.419832    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-560eaa-d8l5q"]
Nov 02 14:17:06.887318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.887283    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"
Nov 02 14:17:06.887776 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.887282    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26" event={"ID":"cf53cdb1-5caf-491f-b3ec-c6136f137949","Type":"ContainerDied","Data":"53581f16a8aa5d0ce52176503e8b8014675b904a8f7d91751dcf041a9ab7aa51"}
Nov 02 14:17:06.887776 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.887432    5572 scope.go:122] "RemoveContainer" containerID="4b49ee2b6528e2ce8c8a55ea3628ecf05067cc5738c5ce55942d2fc51ac9d3a6"
Nov 02 14:17:06.888925 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.888838    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"
Nov 02 14:17:06.888925 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.888843    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv" event={"ID":"cd56285b-33f0-4b90-b725-8fbd0b6eefb7","Type":"ContainerDied","Data":"c4c889bfa743693e8aa006638ab529ebcfd0745c050213db5c99bf1aeb263f63"}
Nov 02 14:17:06.896989 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.896970    5572 scope.go:122] "RemoveContainer" containerID="9ce575d2c32b32697ae5f5984ea700af98427638cbfb6a62cc07ea7e1ff634b3"
Nov 02 14:17:06.916001 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.915976    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"]
Nov 02 14:17:06.922996 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.922972    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-560eaa-76fd8c76dc-7gr26"]
Nov 02 14:17:06.934998 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.934939    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"]
Nov 02 14:17:06.941640 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:06.941619    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-560eaa-67cdb9f86c-kqjvv"]
Nov 02 14:17:07.541490 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:07.541454    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" path="/var/lib/kubelet/pods/2e5bc480-40e1-4021-9a5d-bc2e8abfbbac/volumes"
Nov 02 14:17:07.541856 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:07.541842    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd56285b-33f0-4b90-b725-8fbd0b6eefb7" path="/var/lib/kubelet/pods/cd56285b-33f0-4b90-b725-8fbd0b6eefb7/volumes"
Nov 02 14:17:07.542163 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:07.542151    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf53cdb1-5caf-491f-b3ec-c6136f137949" path="/var/lib/kubelet/pods/cf53cdb1-5caf-491f-b3ec-c6136f137949/volumes"
Nov 02 14:17:13.518791 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:13.518757    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"]
Nov 02 14:17:13.519290 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:13.519016    5572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy" containerID="cri-o://31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153" gracePeriod=30
Nov 02 14:17:14.022051 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:14.022023    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"]
Nov 02 14:17:14.026523 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:14.026498    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b252813-qgtgb"]
Nov 02 14:17:15.541456 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:15.541425    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d452288e-bd2e-40ca-b201-e17985be9e13" path="/var/lib/kubelet/pods/d452288e-bd2e-40ca-b201-e17985be9e13/volumes"
Nov 02 14:17:18.766992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.766969    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:17:18.861267 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861240    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc51337-7116-48ac-af1a-d856a8edde40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc51337-7116-48ac-af1a-d856a8edde40-istiod-ca-cert\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861438 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861330    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-data\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861520 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861491    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-credential-socket\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861531    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-data" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:18.861641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861539    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-envoy\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861641 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861627    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edc51337-7116-48ac-af1a-d856a8edde40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc51337-7116-48ac-af1a-d856a8edde40-istio-podinfo\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861812 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861651    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-socket\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861812 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861669    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-kube-api-access-n4zgp\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-kube-api-access-n4zgp\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861812 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861689    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-certs\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861812 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861732    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-istio-token\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-istio-token\") pod \"edc51337-7116-48ac-af1a-d856a8edde40\" (UID: \"edc51337-7116-48ac-af1a-d856a8edde40\") "
Nov 02 14:17:18.861812 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861768    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc51337-7116-48ac-af1a-d856a8edde40-istiod-ca-cert" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:18.861812 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861794    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-credential-socket" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:18.862133 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.861914    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-socket" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:18.862133 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.862073    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-certs" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:18.862133 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.862087    5572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc51337-7116-48ac-af1a-d856a8edde40-istiod-ca-cert\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.862133 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.862106    5572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-data\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.862133 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.862116    5572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-credential-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.862133 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.862126    5572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-socket\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.863924 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.863896    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-kube-api-access-n4zgp" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "kube-api-access-n4zgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:18.864174 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.864154    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-istio-token" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:18.864275 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.864260    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-envoy" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:18.864316 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.864285    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edc51337-7116-48ac-af1a-d856a8edde40-istio-podinfo" pod "edc51337-7116-48ac-af1a-d856a8edde40" (UID: "edc51337-7116-48ac-af1a-d856a8edde40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:17:18.962767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.962737    5572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-istio-envoy\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.962767 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.962767    5572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc51337-7116-48ac-af1a-d856a8edde40-istio-podinfo\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.962971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.962779    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4zgp\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-kube-api-access-n4zgp\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.962971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.962790    5572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc51337-7116-48ac-af1a-d856a8edde40-workload-certs\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.962971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.962801    5572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc51337-7116-48ac-af1a-d856a8edde40-istio-token\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:18.967890 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.967861    5572 generic.go:356] "Generic (PLEG): container finished" podID="edc51337-7116-48ac-af1a-d856a8edde40" containerID="31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153" exitCode=0
Nov 02 14:17:18.968035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.967914    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"
Nov 02 14:17:18.968035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.967937    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" event={"ID":"edc51337-7116-48ac-af1a-d856a8edde40","Type":"ContainerDied","Data":"31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153"}
Nov 02 14:17:18.968035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.967994    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z" event={"ID":"edc51337-7116-48ac-af1a-d856a8edde40","Type":"ContainerDied","Data":"609b7543965ec91182cadf143366805407df046c3389aebd6a897df716244059"}
Nov 02 14:17:18.968035 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.968023    5572 scope.go:122] "RemoveContainer" containerID="31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153"
Nov 02 14:17:18.977692 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.977670    5572 scope.go:122] "RemoveContainer" containerID="5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d"
Nov 02 14:17:18.985546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.985531    5572 scope.go:122] "RemoveContainer" containerID="31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153"
Nov 02 14:17:18.985920 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:17:18.985901    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153\": container with ID starting with 31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153 not found: ID does not exist" containerID="31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153"
Nov 02 14:17:18.986046 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.985929    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153"} err="failed to get container status \"31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153\": rpc error: code = NotFound desc = could not find container \"31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153\": container with ID starting with 31fce5a18fb6b9792984282c325693884188a9c2bf5cce5b2d87a33b2ab22153 not found: ID does not exist"
Nov 02 14:17:18.986046 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.985945    5572 scope.go:122] "RemoveContainer" containerID="5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d"
Nov 02 14:17:18.986202 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:17:18.986184    5572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d\": container with ID starting with 5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d not found: ID does not exist" containerID="5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d"
Nov 02 14:17:18.986240 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.986209    5572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d"} err="failed to get container status \"5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d\": rpc error: code = NotFound desc = could not find container \"5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d\": container with ID starting with 5a152c340d8b01bde6e57f648eab977ba986066a6ffce95d5d61d0bc861bef0d not found: ID does not exist"
Nov 02 14:17:18.990594 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.990547    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"]
Nov 02 14:17:18.997587 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:18.997554    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-560eaa-openshift-default-544b6cf89c-vmq5z"]
Nov 02 14:17:19.541992 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:19.541957    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc51337-7116-48ac-af1a-d856a8edde40" path="/var/lib/kubelet/pods/edc51337-7116-48ac-af1a-d856a8edde40/volumes"
Nov 02 14:17:21.432854 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:17:21.432818    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:17:25.014244 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014213    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"]
Nov 02 14:17:25.014873 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014835    5572 cpu_manager.go:414] "containerMap: removing container" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:17:25.014873 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014854    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:17:25.014873 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014867    5572 cpu_manager.go:414] "containerMap: removing container" podUID="2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" containerName="revoke-keys"
Nov 02 14:17:25.014873 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014874    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" containerName="revoke-keys"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014905    5572 cpu_manager.go:414] "containerMap: removing container" podUID="cf53cdb1-5caf-491f-b3ec-c6136f137949" containerName="payload-pre-processing"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014915    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf53cdb1-5caf-491f-b3ec-c6136f137949" containerName="payload-pre-processing"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014935    5572 cpu_manager.go:414] "containerMap: removing container" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014942    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014960    5572 cpu_manager.go:414] "containerMap: removing container" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014967    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014981    5572 cpu_manager.go:414] "containerMap: removing container" podUID="cd56285b-33f0-4b90-b725-8fbd0b6eefb7" containerName="payload-processing"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.014988    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd56285b-33f0-4b90-b725-8fbd0b6eefb7" containerName="payload-processing"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.015090    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5bc480-40e1-4021-9a5d-bc2e8abfbbac" containerName="revoke-keys"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.015107    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf53cdb1-5caf-491f-b3ec-c6136f137949" containerName="payload-pre-processing"
Nov 02 14:17:25.015114 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.015117    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy"
Nov 02 14:17:25.015672 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.015129    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc51337-7116-48ac-af1a-d856a8edde40" containerName="istio-proxy"
Nov 02 14:17:25.015672 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.015139    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd56285b-33f0-4b90-b725-8fbd0b6eefb7" containerName="payload-processing"
Nov 02 14:17:25.015672 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.015147    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeb9b33-76d5-4af0-94b0-1bc0ae6f7fe7" containerName="cleanup"
Nov 02 14:17:25.019816 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.019795    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.027527 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.027341    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9c58\""
Nov 02 14:17:25.027527 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.027410    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:17:25.027963 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.027933    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"]
Nov 02 14:17:25.067847 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.067825    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.067968 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.067858    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.067968 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.067877    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2q9\" (UniqueName: \"kubernetes.io/projected/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-kube-api-access-7h2q9\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.169167 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.169128    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.169167 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.169169    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2q9\" (UniqueName: \"kubernetes.io/projected/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-kube-api-access-7h2q9\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.169430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.169268    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.169986 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.169906    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.171685 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.171664    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.182660 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.182634    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2q9\" (UniqueName: \"kubernetes.io/projected/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-kube-api-access-7h2q9\") pod \"maas-api-revoke-sub-f03629db-8cxk5\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.333969 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.333945    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:25.609318 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:25.609248    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"]
Nov 02 14:17:25.611938 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:17:25.611907    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbe9d03_4623_44f1_8399_fafbaf2f63bf.slice/crio-b09c88851b28aadc95f8a13cf7f78a9db8533082fdf24e280cf802588421f57a WatchSource:0}: Error finding container b09c88851b28aadc95f8a13cf7f78a9db8533082fdf24e280cf802588421f57a: Status 404 returned error can't find the container with id b09c88851b28aadc95f8a13cf7f78a9db8533082fdf24e280cf802588421f57a
Nov 02 14:17:26.024657 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:26.024621    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5" event={"ID":"0cbe9d03-4623-44f1-8399-fafbaf2f63bf","Type":"ContainerStarted","Data":"d9069140b600e22c871498259c7ea77ac10d3d2c9d42e2c32607f37bf478a5fe"}
Nov 02 14:17:26.025050 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:26.024685    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5" event={"ID":"0cbe9d03-4623-44f1-8399-fafbaf2f63bf","Type":"ContainerStarted","Data":"b09c88851b28aadc95f8a13cf7f78a9db8533082fdf24e280cf802588421f57a"}
Nov 02 14:17:26.045598 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:26.045541    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5" podStartSLOduration=2.04552901 podStartE2EDuration="2.04552901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:26.043228027 +0000 UTC m=+2022.621316895" watchObservedRunningTime="2026-11-02 14:17:26.04552901 +0000 UTC m=+2022.623617878"
Nov 02 14:17:27.030463 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:27.030430    5572 generic.go:356] "Generic (PLEG): container finished" podID="0cbe9d03-4623-44f1-8399-fafbaf2f63bf" containerID="d9069140b600e22c871498259c7ea77ac10d3d2c9d42e2c32607f37bf478a5fe" exitCode=0
Nov 02 14:17:27.030876 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:27.030503    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5" event={"ID":"0cbe9d03-4623-44f1-8399-fafbaf2f63bf","Type":"ContainerDied","Data":"d9069140b600e22c871498259c7ea77ac10d3d2c9d42e2c32607f37bf478a5fe"}
Nov 02 14:17:28.170746 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.170723    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:28.199647 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.199622    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-xks-ca\") pod \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") "
Nov 02 14:17:28.199774 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.199672    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-openshift-ca\") pod \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") "
Nov 02 14:17:28.199774 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.199700    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-kube-api-access-7h2q9\" (UniqueName: \"kubernetes.io/projected/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-kube-api-access-7h2q9\") pod \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\" (UID: \"0cbe9d03-4623-44f1-8399-fafbaf2f63bf\") "
Nov 02 14:17:28.200407 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.200332    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-openshift-ca" pod "0cbe9d03-4623-44f1-8399-fafbaf2f63bf" (UID: "0cbe9d03-4623-44f1-8399-fafbaf2f63bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:28.202046 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.202019    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-kube-api-access-7h2q9" pod "0cbe9d03-4623-44f1-8399-fafbaf2f63bf" (UID: "0cbe9d03-4623-44f1-8399-fafbaf2f63bf"). InnerVolumeSpecName "kube-api-access-7h2q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:28.202144 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.202042    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-xks-ca" pod "0cbe9d03-4623-44f1-8399-fafbaf2f63bf" (UID: "0cbe9d03-4623-44f1-8399-fafbaf2f63bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:17:28.301110 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.301046    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:28.301110 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.301070    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:28.301110 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:28.301081    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2q9\" (UniqueName: \"kubernetes.io/projected/0cbe9d03-4623-44f1-8399-fafbaf2f63bf-kube-api-access-7h2q9\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:29.041882 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:29.041848    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5"
Nov 02 14:17:29.042060 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:29.041894    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8cxk5" event={"ID":"0cbe9d03-4623-44f1-8399-fafbaf2f63bf","Type":"ContainerDied","Data":"b09c88851b28aadc95f8a13cf7f78a9db8533082fdf24e280cf802588421f57a"}
Nov 02 14:17:29.042060 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:29.041938    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09c88851b28aadc95f8a13cf7f78a9db8533082fdf24e280cf802588421f57a"
Nov 02 14:17:41.015335 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.015303    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"]
Nov 02 14:17:41.015796 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.015780    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe9d03-4623-44f1-8399-fafbaf2f63bf" containerName="revoke-keys"
Nov 02 14:17:41.015858 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.015800    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe9d03-4623-44f1-8399-fafbaf2f63bf" containerName="revoke-keys"
Nov 02 14:17:41.016117 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.015912    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe9d03-4623-44f1-8399-fafbaf2f63bf" containerName="revoke-keys"
Nov 02 14:17:41.019044 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.019029    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.027692 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.027671    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:17:41.027977 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.027956    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9c58\""
Nov 02 14:17:41.030021 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.029987    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"]
Nov 02 14:17:41.174221 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.174193    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4dc\" (UniqueName: \"kubernetes.io/projected/0a7377ec-ca45-44fb-902b-521cb4509e23-kube-api-access-lk4dc\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.174414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.174241    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.174414 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.174263    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.274863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.274789    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.274863 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.274822    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.275074 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.274904    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4dc\" (UniqueName: \"kubernetes.io/projected/0a7377ec-ca45-44fb-902b-521cb4509e23-kube-api-access-lk4dc\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.275606 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.275555    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.277321 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.277300    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.288312 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.288290    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4dc\" (UniqueName: \"kubernetes.io/projected/0a7377ec-ca45-44fb-902b-521cb4509e23-kube-api-access-lk4dc\") pod \"maas-api-revoke-sub-0165d7d2-885jl\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.333289 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.333265    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:41.610077 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:41.610053    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"]
Nov 02 14:17:41.612621 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:17:41.612592    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7377ec_ca45_44fb_902b_521cb4509e23.slice/crio-20b5be056e0fa3c34516cc7c666d6223ef9e03007dbb53ba709d9866b1bcd25e WatchSource:0}: Error finding container 20b5be056e0fa3c34516cc7c666d6223ef9e03007dbb53ba709d9866b1bcd25e: Status 404 returned error can't find the container with id 20b5be056e0fa3c34516cc7c666d6223ef9e03007dbb53ba709d9866b1bcd25e
Nov 02 14:17:42.132383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:42.132343    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl" event={"ID":"0a7377ec-ca45-44fb-902b-521cb4509e23","Type":"ContainerStarted","Data":"0a8f9acd41ab3904d5cede3a151c4c654cb4d16db356104f9f767c4a0641015d"}
Nov 02 14:17:42.132823 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:42.132412    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl" event={"ID":"0a7377ec-ca45-44fb-902b-521cb4509e23","Type":"ContainerStarted","Data":"20b5be056e0fa3c34516cc7c666d6223ef9e03007dbb53ba709d9866b1bcd25e"}
Nov 02 14:17:42.151690 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:42.151608    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl" podStartSLOduration=2.151595444 podStartE2EDuration="2.151595444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:42.150903381 +0000 UTC m=+2038.728992250" watchObservedRunningTime="2026-11-02 14:17:42.151595444 +0000 UTC m=+2038.729684311"
Nov 02 14:17:43.138576 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:43.138527    5572 generic.go:356] "Generic (PLEG): container finished" podID="0a7377ec-ca45-44fb-902b-521cb4509e23" containerID="0a8f9acd41ab3904d5cede3a151c4c654cb4d16db356104f9f767c4a0641015d" exitCode=0
Nov 02 14:17:43.138962 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:43.138609    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl" event={"ID":"0a7377ec-ca45-44fb-902b-521cb4509e23","Type":"ContainerDied","Data":"0a8f9acd41ab3904d5cede3a151c4c654cb4d16db356104f9f767c4a0641015d"}
Nov 02 14:17:44.276524 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.276500    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:44.301029 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.301001    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7377ec-ca45-44fb-902b-521cb4509e23-kube-api-access-lk4dc\" (UniqueName: \"kubernetes.io/projected/0a7377ec-ca45-44fb-902b-521cb4509e23-kube-api-access-lk4dc\") pod \"0a7377ec-ca45-44fb-902b-521cb4509e23\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") "
Nov 02 14:17:44.301178 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.301051    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-openshift-ca\") pod \"0a7377ec-ca45-44fb-902b-521cb4509e23\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") "
Nov 02 14:17:44.301178 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.301089    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-xks-ca\") pod \"0a7377ec-ca45-44fb-902b-521cb4509e23\" (UID: \"0a7377ec-ca45-44fb-902b-521cb4509e23\") "
Nov 02 14:17:44.301447 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.301424    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-openshift-ca" pod "0a7377ec-ca45-44fb-902b-521cb4509e23" (UID: "0a7377ec-ca45-44fb-902b-521cb4509e23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:44.303364 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.303338    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7377ec-ca45-44fb-902b-521cb4509e23-kube-api-access-lk4dc" pod "0a7377ec-ca45-44fb-902b-521cb4509e23" (UID: "0a7377ec-ca45-44fb-902b-521cb4509e23"). InnerVolumeSpecName "kube-api-access-lk4dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:44.303511 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.303488    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-xks-ca" pod "0a7377ec-ca45-44fb-902b-521cb4509e23" (UID: "0a7377ec-ca45-44fb-902b-521cb4509e23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:17:44.402444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.402380    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4dc\" (UniqueName: \"kubernetes.io/projected/0a7377ec-ca45-44fb-902b-521cb4509e23-kube-api-access-lk4dc\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:44.402444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.402403    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:44.402444 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:44.402414    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7377ec-ca45-44fb-902b-521cb4509e23-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:45.149520 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:45.149494    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl"
Nov 02 14:17:45.149746 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:45.149586    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-885jl" event={"ID":"0a7377ec-ca45-44fb-902b-521cb4509e23","Type":"ContainerDied","Data":"20b5be056e0fa3c34516cc7c666d6223ef9e03007dbb53ba709d9866b1bcd25e"}
Nov 02 14:17:45.149746 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:45.149641    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b5be056e0fa3c34516cc7c666d6223ef9e03007dbb53ba709d9866b1bcd25e"
Nov 02 14:17:45.398214 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:45.398186    5572 scope.go:122] "RemoveContainer" containerID="402bed27cd5b5ab90527fe84b58532931dbec63f9cc395395269838646ef9cc9"
Nov 02 14:17:51.429385 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:17:51.429348    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:17:51.828505 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.828467    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"]
Nov 02 14:17:51.829149 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.829125    5572 cpu_manager.go:414] "containerMap: removing container" podUID="0a7377ec-ca45-44fb-902b-521cb4509e23" containerName="revoke-keys"
Nov 02 14:17:51.829253 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.829151    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7377ec-ca45-44fb-902b-521cb4509e23" containerName="revoke-keys"
Nov 02 14:17:51.829369 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.829273    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7377ec-ca45-44fb-902b-521cb4509e23" containerName="revoke-keys"
Nov 02 14:17:51.834104 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.834086    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.841296 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.841274    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"]
Nov 02 14:17:51.855907 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.851724    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9c58\""
Nov 02 14:17:51.855907 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.851773    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:17:51.863592 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.863543    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsz62\" (UniqueName: \"kubernetes.io/projected/075c6c32-230a-4dd5-9850-0635657ce0ad-kube-api-access-bsz62\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.863718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.863651    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.863718 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.863676    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.965339 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.965302    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsz62\" (UniqueName: \"kubernetes.io/projected/075c6c32-230a-4dd5-9850-0635657ce0ad-kube-api-access-bsz62\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.965512 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.965352    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.965512 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.965393    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.966058 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.966036    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.967906 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.967882    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:51.977937 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:51.977914    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsz62\" (UniqueName: \"kubernetes.io/projected/075c6c32-230a-4dd5-9850-0635657ce0ad-kube-api-access-bsz62\") pod \"maas-api-revoke-sub-6b19338f-v76vv\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:52.151437 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:52.151376    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:52.460028 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:52.459836    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"]
Nov 02 14:17:52.462459 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:17:52.462432    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075c6c32_230a_4dd5_9850_0635657ce0ad.slice/crio-50bf83cd84812fb360a2fcc6a9082f629f470a31d728747bc517350ed0f3c76a WatchSource:0}: Error finding container 50bf83cd84812fb360a2fcc6a9082f629f470a31d728747bc517350ed0f3c76a: Status 404 returned error can't find the container with id 50bf83cd84812fb360a2fcc6a9082f629f470a31d728747bc517350ed0f3c76a
Nov 02 14:17:53.191901 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:53.191869    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv" event={"ID":"075c6c32-230a-4dd5-9850-0635657ce0ad","Type":"ContainerStarted","Data":"2aa4fff017602ef021a01ca20401304c6ca9459c18fcba906bcebca6890a9714"}
Nov 02 14:17:53.192066 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:53.191921    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv" event={"ID":"075c6c32-230a-4dd5-9850-0635657ce0ad","Type":"ContainerStarted","Data":"50bf83cd84812fb360a2fcc6a9082f629f470a31d728747bc517350ed0f3c76a"}
Nov 02 14:17:53.211834 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:53.211791    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv" podStartSLOduration=2.211774056 podStartE2EDuration="2.211774056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:53.210225386 +0000 UTC m=+2049.788314315" watchObservedRunningTime="2026-11-02 14:17:53.211774056 +0000 UTC m=+2049.789862923"
Nov 02 14:17:54.198721 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:54.198685    5572 generic.go:356] "Generic (PLEG): container finished" podID="075c6c32-230a-4dd5-9850-0635657ce0ad" containerID="2aa4fff017602ef021a01ca20401304c6ca9459c18fcba906bcebca6890a9714" exitCode=0
Nov 02 14:17:54.199113 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:54.198756    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv" event={"ID":"075c6c32-230a-4dd5-9850-0635657ce0ad","Type":"ContainerDied","Data":"2aa4fff017602ef021a01ca20401304c6ca9459c18fcba906bcebca6890a9714"}
Nov 02 14:17:55.341028 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.341002    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:55.401983 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.401954    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075c6c32-230a-4dd5-9850-0635657ce0ad-kube-api-access-bsz62\" (UniqueName: \"kubernetes.io/projected/075c6c32-230a-4dd5-9850-0635657ce0ad-kube-api-access-bsz62\") pod \"075c6c32-230a-4dd5-9850-0635657ce0ad\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") "
Nov 02 14:17:55.402118 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.402066    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-openshift-ca\") pod \"075c6c32-230a-4dd5-9850-0635657ce0ad\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") "
Nov 02 14:17:55.402118 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.402099    5572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-xks-ca\") pod \"075c6c32-230a-4dd5-9850-0635657ce0ad\" (UID: \"075c6c32-230a-4dd5-9850-0635657ce0ad\") "
Nov 02 14:17:55.402410 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.402385    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-openshift-ca" pod "075c6c32-230a-4dd5-9850-0635657ce0ad" (UID: "075c6c32-230a-4dd5-9850-0635657ce0ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:55.404198 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.404167    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075c6c32-230a-4dd5-9850-0635657ce0ad-kube-api-access-bsz62" pod "075c6c32-230a-4dd5-9850-0635657ce0ad" (UID: "075c6c32-230a-4dd5-9850-0635657ce0ad"). InnerVolumeSpecName "kube-api-access-bsz62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:55.404283 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.404246    5572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-xks-ca" pod "075c6c32-230a-4dd5-9850-0635657ce0ad" (UID: "075c6c32-230a-4dd5-9850-0635657ce0ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:17:55.503580 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.503488    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-openshift-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:55.503580 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.503513    5572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075c6c32-230a-4dd5-9850-0635657ce0ad-maas-api-xks-ca\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:55.503580 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:55.503523    5572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsz62\" (UniqueName: \"kubernetes.io/projected/075c6c32-230a-4dd5-9850-0635657ce0ad-kube-api-access-bsz62\") on node \"ip-10-0-128-62.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 14:17:56.209227 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:56.209195    5572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv"
Nov 02 14:17:56.209227 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:56.209223    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-v76vv" event={"ID":"075c6c32-230a-4dd5-9850-0635657ce0ad","Type":"ContainerDied","Data":"50bf83cd84812fb360a2fcc6a9082f629f470a31d728747bc517350ed0f3c76a"}
Nov 02 14:17:56.209664 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:17:56.209271    5572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50bf83cd84812fb360a2fcc6a9082f629f470a31d728747bc517350ed0f3c76a"
Nov 02 14:18:21.428151 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:18:21.428113    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:18:23.549707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:23.549627    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sk6cs_852b251e-7221-460c-a861-0b6e71ac2571/manager/0.log"
Nov 02 14:18:23.744333 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:23.744305    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vt89d_0c20ef56-0297-49b4-a716-f03385219dc0/manager/0.log"
Nov 02 14:18:24.190487 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:24.190457    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b76f98b8f-hmbs8_332e62e8-9f90-4f0b-812b-f7eff688870f/manager/0.log"
Nov 02 14:18:24.605715 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:24.605690    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fj5wt_69a2d9a3-6280-4635-b146-7fa5b060f049/manager/0.log"
Nov 02 14:18:25.941041 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:25.941012    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-885jl_0a7377ec-ca45-44fb-902b-521cb4509e23/revoke-keys/0.log"
Nov 02 14:18:26.105413 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:26.105384    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-qhwbm_846a8578-64ec-443b-bf28-2af9dfac289b/revoke-keys/0.log"
Nov 02 14:18:26.281206 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:26.281133    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-t2bmc_65cbe482-abe8-4ef0-aa6f-b2c4087b4e68/revoke-keys/0.log"
Nov 02 14:18:26.459743 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:26.459706    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-55f42e7d-xbrc4_c910f7a0-b1eb-4166-97cd-931842187df6/revoke-keys/0.log"
Nov 02 14:18:26.643793 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:26.643760    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v76vv_075c6c32-230a-4dd5-9850-0635657ce0ad/revoke-keys/0.log"
Nov 02 14:18:26.816082 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:26.816045    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8cxk5_0cbe9d03-4623-44f1-8399-fafbaf2f63bf/revoke-keys/0.log"
Nov 02 14:18:26.991152 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:26.991076    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-p9hzc_1da4efa5-63bb-45bc-90fc-0ceca7c2e04e/postgres/0.log"
Nov 02 14:18:30.021487 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:30.021455    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"]
Nov 02 14:18:30.024182 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:30.024126    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t2bmc"]
Nov 02 14:18:30.620080 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:30.620047    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-852sj_39308b2f-8194-4827-8f2f-b0be25d372e6/discovery/0.log"
Nov 02 14:18:30.823543 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:30.823513    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fb985f76c-7xxff_1cef31c7-8dba-4891-940f-1ef1d998854b/kube-auth-proxy/0.log"
Nov 02 14:18:31.345605 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:31.345578    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-bnw6z_f7165380-1253-4546-b0bf-f80b2df80754/istio-proxy/0.log"
Nov 02 14:18:31.541536 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:31.541504    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cbe482-abe8-4ef0-aa6f-b2c4087b4e68" path="/var/lib/kubelet/pods/65cbe482-abe8-4ef0-aa6f-b2c4087b4e68/volumes"
Nov 02 14:18:31.698256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:31.698182    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-drgtc_b508b5b5-3d14-4bc5-8462-a45ec55b2b49/istio-proxy/0.log"
Nov 02 14:18:32.381926 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:32.381887    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-l7sbn_fc2035df-6646-4894-946b-6b38c5d60f77/istio-proxy/0.log"
Nov 02 14:18:33.055170 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:33.055133    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kbtw9_ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7/payload-pre-processing/0.log"
Nov 02 14:18:33.220374 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:33.220347    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:18:34.456809 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:34.456772    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q595n_6b462672-55fe-42b8-b39f-52b0de99145a/main/0.log"
Nov 02 14:18:40.546531 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:40.546492    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2_87666c0f-6310-470a-8242-b4345b50acbf/main/0.log"
Nov 02 14:18:45.023256 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:45.023217    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"]
Nov 02 14:18:45.028828 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:45.028800    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhwbm"]
Nov 02 14:18:45.429775 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:45.429739    5572 scope.go:122] "RemoveContainer" containerID="00aca135f1d079072d4ed71ad1c91ae3e05cf278a328440bee9b192388a64c97"
Nov 02 14:18:45.438479 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:45.438454    5572 scope.go:122] "RemoveContainer" containerID="6be1adf627d693f519245dca9843764216b78de9a96a76bc2eccb46fbd4914fd"
Nov 02 14:18:45.540653 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:45.540623    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846a8578-64ec-443b-bf28-2af9dfac289b" path="/var/lib/kubelet/pods/846a8578-64ec-443b-bf28-2af9dfac289b/volumes"
Nov 02 14:18:50.677768 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:50.677736    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sk6cs_852b251e-7221-460c-a861-0b6e71ac2571/manager/0.log"
Nov 02 14:18:50.869954 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:50.869921    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-vt89d_0c20ef56-0297-49b4-a716-f03385219dc0/manager/0.log"
Nov 02 14:18:51.297971 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:51.297939    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b76f98b8f-hmbs8_332e62e8-9f90-4f0b-812b-f7eff688870f/manager/0.log"
Nov 02 14:18:51.428869 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:18:51.428821    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:18:51.688891 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:51.688853    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fj5wt_69a2d9a3-6280-4635-b146-7fa5b060f049/manager/0.log"
Nov 02 14:18:52.767997 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:52.767968    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-885jl_0a7377ec-ca45-44fb-902b-521cb4509e23/revoke-keys/0.log"
Nov 02 14:18:52.993541 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:52.993512    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-55f42e7d-xbrc4_c910f7a0-b1eb-4166-97cd-931842187df6/revoke-keys/0.log"
Nov 02 14:18:53.162623 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:53.162583    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-v76vv_075c6c32-230a-4dd5-9850-0635657ce0ad/revoke-keys/0.log"
Nov 02 14:18:53.334786 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:53.334756    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8cxk5_0cbe9d03-4623-44f1-8399-fafbaf2f63bf/revoke-keys/0.log"
Nov 02 14:18:53.497239 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:53.497165    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-p9hzc_1da4efa5-63bb-45bc-90fc-0ceca7c2e04e/postgres/0.log"
Nov 02 14:18:56.532075 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:56.532045    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-852sj_39308b2f-8194-4827-8f2f-b0be25d372e6/discovery/0.log"
Nov 02 14:18:56.711529 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:56.711504    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fb985f76c-7xxff_1cef31c7-8dba-4891-940f-1ef1d998854b/kube-auth-proxy/0.log"
Nov 02 14:18:57.263974 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:57.263932    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-bnw6z_f7165380-1253-4546-b0bf-f80b2df80754/istio-proxy/0.log"
Nov 02 14:18:57.626324 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:57.626299    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-drgtc_b508b5b5-3d14-4bc5-8462-a45ec55b2b49/istio-proxy/0.log"
Nov 02 14:18:58.331896 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:58.331863    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-l7sbn_fc2035df-6646-4894-946b-6b38c5d60f77/istio-proxy/0.log"
Nov 02 14:18:59.073925 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:59.073889    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kbtw9_ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7/payload-pre-processing/0.log"
Nov 02 14:18:59.243373 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:18:59.243350    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:19:00.125865 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:00.125830    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q595n_6b462672-55fe-42b8-b39f-52b0de99145a/main/0.log"
Nov 02 14:19:01.211017 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:01.210981    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9l8j2_87666c0f-6310-470a-8242-b4345b50acbf/main/0.log"
Nov 02 14:19:05.077032 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:05.076996    5572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"]
Nov 02 14:19:05.098645 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:05.093646    5572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55f42e7d-xbrc4"]
Nov 02 14:19:05.541602 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:05.541502    5572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c910f7a0-b1eb-4166-97cd-931842187df6" path="/var/lib/kubelet/pods/c910f7a0-b1eb-4166-97cd-931842187df6/volumes"
Nov 02 14:19:21.429034 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:19:21.428995    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:19:45.460642 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:45.460610    5572 scope.go:122] "RemoveContainer" containerID="dc396689e0a9eb32c5ae705f7d2ad5a73071e70ec560fabeed849d3093a36b1e"
Nov 02 14:19:45.469026 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:45.469009    5572 scope.go:122] "RemoveContainer" containerID="5e5963e7447723ded7f5b54b9bb628a79f002689dcd473680beff6730999d9c1"
Nov 02 14:19:45.476648 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:19:45.476631    5572 scope.go:122] "RemoveContainer" containerID="b372a3ca498ab38252c46aae410ebc674dcbad2b7eb6d370e6e4934aac2807e9"
Nov 02 14:19:51.429814 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:19:51.429777    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:20:21.429095 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:20:21.429054    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:20:26.804368 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:20:26.804336    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b76f98b8f-hmbs8_332e62e8-9f90-4f0b-812b-f7eff688870f/manager/0.log"
Nov 02 14:20:27.059825 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:20:27.059796    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b76f98b8f-hmbs8_332e62e8-9f90-4f0b-812b-f7eff688870f/manager/0.log"
Nov 02 14:20:27.747810 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:20:27.747779    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:20:27.904715 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:20:27.904687    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:20:45.500784 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:20:45.500750    5572 scope.go:122] "RemoveContainer" containerID="6d8b1a03af04f9969f4231020213fb0b4a710836a2c4d5b85cf3433625bf96da"
Nov 02 14:20:45.509388 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:20:45.509367    5572 scope.go:122] "RemoveContainer" containerID="0257f6d682c111d39e77c9921c808b74954f48d9ac5282169594986cadd6f0d2"
Nov 02 14:20:51.429858 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:20:51.429819    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:21:11.249690 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:11.249657    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b76f98b8f-hmbs8_332e62e8-9f90-4f0b-812b-f7eff688870f/manager/0.log"
Nov 02 14:21:11.502449 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:11.502366    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b76f98b8f-hmbs8_332e62e8-9f90-4f0b-812b-f7eff688870f/manager/0.log"
Nov 02 14:21:12.221827 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.221794    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:21:12.395738 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.395708    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:21:12.720546 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.720513    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g76p8/must-gather-vkbv6"]
Nov 02 14:21:12.720941 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.720927    5572 cpu_manager.go:414] "containerMap: removing container" podUID="075c6c32-230a-4dd5-9850-0635657ce0ad" containerName="revoke-keys"
Nov 02 14:21:12.720991 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.720942    5572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075c6c32-230a-4dd5-9850-0635657ce0ad" containerName="revoke-keys"
Nov 02 14:21:12.721031 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.721008    5572 memory_manager.go:365] "RemoveStaleState removing state" podUID="075c6c32-230a-4dd5-9850-0635657ce0ad" containerName="revoke-keys"
Nov 02 14:21:12.724387 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.724371    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:12.735897 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.735871    5572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g76p8\"/\"default-dockercfg-f4qh7\""
Nov 02 14:21:12.736880 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.736865    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g76p8\"/\"openshift-service-ca.crt\""
Nov 02 14:21:12.737014 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.737001    5572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g76p8\"/\"kube-root-ca.crt\""
Nov 02 14:21:12.750405 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.750382    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g76p8/must-gather-vkbv6"]
Nov 02 14:21:12.763880 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.763850    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8a7e660e-1aed-443d-bb9b-2c605ccfd509-must-gather-output\") pod \"must-gather-vkbv6\" (UID: \"8a7e660e-1aed-443d-bb9b-2c605ccfd509\") " pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:12.764018 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.763949    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzqv\" (UniqueName: \"kubernetes.io/projected/8a7e660e-1aed-443d-bb9b-2c605ccfd509-kube-api-access-mzzqv\") pod \"must-gather-vkbv6\" (UID: \"8a7e660e-1aed-443d-bb9b-2c605ccfd509\") " pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:12.864650 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.864595    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8a7e660e-1aed-443d-bb9b-2c605ccfd509-must-gather-output\") pod \"must-gather-vkbv6\" (UID: \"8a7e660e-1aed-443d-bb9b-2c605ccfd509\") " pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:12.864838 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.864676    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzqv\" (UniqueName: \"kubernetes.io/projected/8a7e660e-1aed-443d-bb9b-2c605ccfd509-kube-api-access-mzzqv\") pod \"must-gather-vkbv6\" (UID: \"8a7e660e-1aed-443d-bb9b-2c605ccfd509\") " pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:12.864953 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.864932    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8a7e660e-1aed-443d-bb9b-2c605ccfd509-must-gather-output\") pod \"must-gather-vkbv6\" (UID: \"8a7e660e-1aed-443d-bb9b-2c605ccfd509\") " pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:12.882973 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:12.882938    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzqv\" (UniqueName: \"kubernetes.io/projected/8a7e660e-1aed-443d-bb9b-2c605ccfd509-kube-api-access-mzzqv\") pod \"must-gather-vkbv6\" (UID: \"8a7e660e-1aed-443d-bb9b-2c605ccfd509\") " pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:13.038383 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:13.038304    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g76p8/must-gather-vkbv6"
Nov 02 14:21:13.312172 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:13.312147    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g76p8/must-gather-vkbv6"]
Nov 02 14:21:13.314790 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:21:13.314762    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a7e660e_1aed_443d_bb9b_2c605ccfd509.slice/crio-9503121830f4ea41d6dbf22df762333b7534e1bbb8b8e4812b0d48e8d2ed7bdc WatchSource:0}: Error finding container 9503121830f4ea41d6dbf22df762333b7534e1bbb8b8e4812b0d48e8d2ed7bdc: Status 404 returned error can't find the container with id 9503121830f4ea41d6dbf22df762333b7534e1bbb8b8e4812b0d48e8d2ed7bdc
Nov 02 14:21:13.316707 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:13.316688    5572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:21:14.163999 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:14.163956    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g76p8/must-gather-vkbv6" event={"ID":"8a7e660e-1aed-443d-bb9b-2c605ccfd509","Type":"ContainerStarted","Data":"887884e85529a4ad149264e7daa23ab3b689ea2868e2f583fa63c9c538c00636"}
Nov 02 14:21:14.164439 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:14.164015    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g76p8/must-gather-vkbv6" event={"ID":"8a7e660e-1aed-443d-bb9b-2c605ccfd509","Type":"ContainerStarted","Data":"9503121830f4ea41d6dbf22df762333b7534e1bbb8b8e4812b0d48e8d2ed7bdc"}
Nov 02 14:21:15.176693 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:15.176658    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g76p8/must-gather-vkbv6" event={"ID":"8a7e660e-1aed-443d-bb9b-2c605ccfd509","Type":"ContainerStarted","Data":"b665880e9b5a2ced4b9b63f04ab70c85694e4443b44742c44b4c6d27258f6bf2"}
Nov 02 14:21:15.195627 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:15.195516    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-g76p8/must-gather-vkbv6" podStartSLOduration=2.472546837 podStartE2EDuration="3.195496921s" totalImagesPullingTime="722.950084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:15.19400827 +0000 UTC m=+2251.772097138" watchObservedRunningTime="2026-11-02 14:21:15.195496921 +0000 UTC m=+2251.773585788"
Nov 02 14:21:15.999278 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:15.999244    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q6z7m_71c1e4ef-71b9-4a62-b841-a21453d83e59/global-pull-secret-syncer/0.log"
Nov 02 14:21:16.220868 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:16.220836    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jt476_70301f22-b14c-4362-a082-15309eafa41b/konnectivity-agent/0.log"
Nov 02 14:21:16.380211 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:16.380178    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-62.us-east-2.compute.internal_b4ea240f21eef62637ef08b4bbcf8f6a/haproxy/0.log"
Nov 02 14:21:21.469943 ip-10-0-128-62 kubenswrapper[5572]: E1102 14:21:21.469872    5572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8gf9j" containerName="extractor"
Nov 02 14:21:24.761015 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:24.760924    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719ae74f-7be3-4208-9462-d98255364534/alertmanager/0.log"
Nov 02 14:21:24.793430 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:24.793400    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719ae74f-7be3-4208-9462-d98255364534/config-reloader/0.log"
Nov 02 14:21:24.838299 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:24.838270    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719ae74f-7be3-4208-9462-d98255364534/kube-rbac-proxy-web/0.log"
Nov 02 14:21:24.872967 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:24.872938    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719ae74f-7be3-4208-9462-d98255364534/kube-rbac-proxy/0.log"
Nov 02 14:21:24.910278 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:24.910252    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719ae74f-7be3-4208-9462-d98255364534/kube-rbac-proxy-metric/0.log"
Nov 02 14:21:24.940391 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:24.940366    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719ae74f-7be3-4208-9462-d98255364534/prom-label-proxy/0.log"
Nov 02 14:21:24.987667 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:24.986419    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719ae74f-7be3-4208-9462-d98255364534/init-config-reloader/0.log"
Nov 02 14:21:25.154505 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.154477    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b5cc8db4b-7gz9c_19f1ad48-874e-4463-b936-0704c67be0fd/metrics-server/0.log"
Nov 02 14:21:25.325852 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.325826    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tczrv_ae0be5eb-c95f-462e-ab68-3e4dffaae950/node-exporter/0.log"
Nov 02 14:21:25.359127 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.359094    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tczrv_ae0be5eb-c95f-462e-ab68-3e4dffaae950/kube-rbac-proxy/0.log"
Nov 02 14:21:25.392320 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.392296    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tczrv_ae0be5eb-c95f-462e-ab68-3e4dffaae950/init-textfile/0.log"
Nov 02 14:21:25.688656 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.682216    5572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"]
Nov 02 14:21:25.697640 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.697596    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.699695 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.698616    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"]
Nov 02 14:21:25.790002 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.789961    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-sys\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.790002 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.790003    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-podres\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.790578 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.790226    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-lib-modules\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.790578 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.790323    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twg5f\" (UniqueName: \"kubernetes.io/projected/4f309035-6f10-4638-93af-9b2097f5ff82-kube-api-access-twg5f\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.790578 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.790382    5572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-proc\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.843394 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.843360    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zpxdr_aa65ead1-60e6-42c4-bc86-ef6df717b751/prometheus-operator/0.log"
Nov 02 14:21:25.874960 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.874932    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zpxdr_aa65ead1-60e6-42c4-bc86-ef6df717b751/kube-rbac-proxy/0.log"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.891260    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-lib-modules\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.891363    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twg5f\" (UniqueName: \"kubernetes.io/projected/4f309035-6f10-4638-93af-9b2097f5ff82-kube-api-access-twg5f\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.891406    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-proc\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.891492    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-sys\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.891533    5572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-podres\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.891692    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-podres\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.891778    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-lib-modules\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.892153    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-proc\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.898579 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.892219    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f309035-6f10-4638-93af-9b2097f5ff82-sys\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.913415 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.913373    5572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twg5f\" (UniqueName: \"kubernetes.io/projected/4f309035-6f10-4638-93af-9b2097f5ff82-kube-api-access-twg5f\") pod \"perf-node-gather-daemonset-wnp5t\" (UID: \"4f309035-6f10-4638-93af-9b2097f5ff82\") " pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:25.941711 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.941626    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85fd9b44-djhqk_eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b/telemeter-client/0.log"
Nov 02 14:21:25.968420 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.968388    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85fd9b44-djhqk_eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b/reload/0.log"
Nov 02 14:21:25.999384 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:25.999352    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85fd9b44-djhqk_eb9d31b8-c96d-44d0-a6c2-ebaf72e0577b/kube-rbac-proxy/0.log"
Nov 02 14:21:26.017590 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.017542    5572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:26.036034 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.036011    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c454df6-wgrrz_8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/thanos-query/0.log"
Nov 02 14:21:26.067605 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.066859    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c454df6-wgrrz_8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/kube-rbac-proxy-web/0.log"
Nov 02 14:21:26.100157 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.100132    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c454df6-wgrrz_8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/kube-rbac-proxy/0.log"
Nov 02 14:21:26.140442 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.140413    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c454df6-wgrrz_8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/prom-label-proxy/0.log"
Nov 02 14:21:26.167865 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.167825    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c454df6-wgrrz_8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/kube-rbac-proxy-rules/0.log"
Nov 02 14:21:26.205350 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.201706    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c454df6-wgrrz_8cfd9f5a-4325-4769-9b66-8ef4dc715b5c/kube-rbac-proxy-metrics/0.log"
Nov 02 14:21:26.298072 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:26.297890    5572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"]
Nov 02 14:21:26.305275 ip-10-0-128-62 kubenswrapper[5572]: W1102 14:21:26.304976    5572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4f309035_6f10_4638_93af_9b2097f5ff82.slice/crio-23c55f5218cf64c2b3da4017e53a982798507c3ed17f4215342ebe0347fb7cf0 WatchSource:0}: Error finding container 23c55f5218cf64c2b3da4017e53a982798507c3ed17f4215342ebe0347fb7cf0: Status 404 returned error can't find the container with id 23c55f5218cf64c2b3da4017e53a982798507c3ed17f4215342ebe0347fb7cf0
Nov 02 14:21:27.285206 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:27.285157    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t" event={"ID":"4f309035-6f10-4638-93af-9b2097f5ff82","Type":"ContainerStarted","Data":"3206e73ce5e4b2b88c0d234960db052811554fa13d6998b79d01e3d48fead5af"}
Nov 02 14:21:27.285644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:27.285227    5572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t" event={"ID":"4f309035-6f10-4638-93af-9b2097f5ff82","Type":"ContainerStarted","Data":"23c55f5218cf64c2b3da4017e53a982798507c3ed17f4215342ebe0347fb7cf0"}
Nov 02 14:21:27.285644 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:27.285260    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:27.307117 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:27.307063    5572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t" podStartSLOduration=2.30704894 podStartE2EDuration="2.30704894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:27.306749513 +0000 UTC m=+2263.884838381" watchObservedRunningTime="2026-11-02 14:21:27.30704894 +0000 UTC m=+2263.885137807"
Nov 02 14:21:28.670709 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:28.670678    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cd84cb498-hktk5_7ce3b8ce-ae26-4d13-a669-b511ecbd75f6/console/0.log"
Nov 02 14:21:30.563326 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:30.563302    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-29msq_582ae8d7-d1b6-4066-a863-a14a5f2eb45f/dns/0.log"
Nov 02 14:21:30.600602 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:30.600571    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-29msq_582ae8d7-d1b6-4066-a863-a14a5f2eb45f/kube-rbac-proxy/0.log"
Nov 02 14:21:30.774061 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:30.774032    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sfqfd_73189526-2e64-4de5-a2a9-06597d293b6b/dns-node-resolver/0.log"
Nov 02 14:21:31.540012 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:31.539979    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxttx_5d8ef4a9-bc67-43d4-a30a-2bcd08bbbeea/node-ca/0.log"
Nov 02 14:21:33.303273 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:33.303242    5572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g76p8/perf-node-gather-daemonset-wnp5t"
Nov 02 14:21:34.553232 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:34.553193    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-852sj_39308b2f-8194-4827-8f2f-b0be25d372e6/discovery/0.log"
Nov 02 14:21:34.582493 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:34.582463    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fb985f76c-7xxff_1cef31c7-8dba-4891-940f-1ef1d998854b/kube-auth-proxy/0.log"
Nov 02 14:21:34.727048 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:34.727015    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-bnw6z_f7165380-1253-4546-b0bf-f80b2df80754/istio-proxy/0.log"
Nov 02 14:21:34.821940 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:34.821872    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-drgtc_b508b5b5-3d14-4bc5-8462-a45ec55b2b49/istio-proxy/0.log"
Nov 02 14:21:34.987978 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:34.987945    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-l7sbn_fc2035df-6646-4894-946b-6b38c5d60f77/istio-proxy/0.log"
Nov 02 14:21:35.131365 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:35.131286    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kbtw9_ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7/payload-pre-processing/0.log"
Nov 02 14:21:35.174526 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:35.174496    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:21:36.821124 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:36.821048    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vpzfd_cf3f6767-bfe3-4c84-b337-e17679488a51/serve-healthcheck-canary/0.log"
Nov 02 14:21:37.693359 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:37.693322    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gf9j_21c18bda-36de-4e16-9ea6-952ace591a1f/kube-rbac-proxy/0.log"
Nov 02 14:21:37.722454 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:37.722427    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gf9j_21c18bda-36de-4e16-9ea6-952ace591a1f/exporter/0.log"
Nov 02 14:21:37.760166 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:37.760134    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8gf9j_21c18bda-36de-4e16-9ea6-952ace591a1f/extractor/0.log"
Nov 02 14:21:38.347212 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:38.347178    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-852sj_39308b2f-8194-4827-8f2f-b0be25d372e6/discovery/0.log"
Nov 02 14:21:38.358867 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:38.358841    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fb985f76c-7xxff_1cef31c7-8dba-4891-940f-1ef1d998854b/kube-auth-proxy/0.log"
Nov 02 14:21:38.423451 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:38.423420    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-bnw6z_f7165380-1253-4546-b0bf-f80b2df80754/istio-proxy/0.log"
Nov 02 14:21:38.462757 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:38.462723    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-drgtc_b508b5b5-3d14-4bc5-8462-a45ec55b2b49/istio-proxy/0.log"
Nov 02 14:21:38.546033 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:38.546003    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-l7sbn_fc2035df-6646-4894-946b-6b38c5d60f77/istio-proxy/0.log"
Nov 02 14:21:38.622388 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:38.622294    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kbtw9_ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7/payload-pre-processing/0.log"
Nov 02 14:21:38.641937 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:38.641915    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"
Nov 02 14:21:40.575054 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:40.575023    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-c559bd947-hztw5_1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc/manager/0.log"
Nov 02 14:21:40.661136 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:40.661111    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-c559bd947-hztw5_1a716fcb-ad2a-41ed-8a1b-aadcdc9491bc/copy-manifests/0.log"
Nov 02 14:21:42.246243 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.243247    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-852sj_39308b2f-8194-4827-8f2f-b0be25d372e6/discovery/0.log"
Nov 02 14:21:42.294386 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.294359    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fb985f76c-7xxff_1cef31c7-8dba-4891-940f-1ef1d998854b/kube-auth-proxy/0.log"
Nov 02 14:21:42.368897 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.368851    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-bnw6z_f7165380-1253-4546-b0bf-f80b2df80754/istio-proxy/0.log"
Nov 02 14:21:42.413458 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.413426    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-drgtc_b508b5b5-3d14-4bc5-8462-a45ec55b2b49/istio-proxy/0.log"
Nov 02 14:21:42.463963 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.463935    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-784f4d6794-g7pn8_83a84319-eb06-4af8-9383-410a49087ea0/manager/0.log"
Nov 02 14:21:42.513671 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.513602    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-l7sbn_fc2035df-6646-4894-946b-6b38c5d60f77/istio-proxy/0.log"
Nov 02 14:21:42.604244 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.604220    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kbtw9_ba2cfc9f-85e6-48c9-b3a9-3f6c168b92b7/payload-pre-processing/0.log"
Nov 02 14:21:42.623242 ip-10-0-128-62 kubenswrapper[5572]: I1102 14:21:42.623193    5572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gws8m_abb68a58-70dd-4487-8da8-542ad36afcea/payload-processing/0.log"